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Glossary  of  Terms  and  Acronyms 


ORGANIZATION 


Alternative 


Analysis 


Analyst 


Client 


CoBP 


One  of  two  or  more  possibilities  that  can  be  chosen  for  addressing, 
improving  or  (re)solving  a  problematic  situation. 

A  set  of  activities  aimed  at  decomposing  (physical,  organisational, 
social)  systems,  problems,  problematic  situations,  operations,  etc., 
into  their  constituent  parts,  and  at  investigating  their  relationships 
and  their  meaning. 

An  individual  who  conducts  a  study,  and  in  this  capacity  designs  the 
stages  of  the  process,  suggests  scientific  ways  to  investigate  and  model 
the  problematic  situation,  the  methodology(ies)  and  method(s)  to  be 
used,  the  workshops  to  be  held,  and  ways  to  report  and  interpret  the 
study’s  outcomes. 

This  term  denotes  four  specific  types  of  stakeholders: 

-  The  sponsor  who  actually  owns  the  study; 

-  The  customer  who  pays  the  bill  for  the  study; 

-  The  decision  maker  who  may  make  decisions  regarding  the 
problematic  situation  or  concerning  the  recommendations  by 
(outcomes  of)  the  study  in  preparation  of  decisions  by  higher 
authority;  and 

-  The  end  user  who  is  ultimately  affected  (either  positively  or 
negatively)  by  the  study  outcomes  and  related  decisions. 

The  various  client  types  together  are  often  referred  to  as  ‘client 
system’  in  the  CoBP. 

Code  of  Best  Practice 


Credibility 

Customer 

Data 

Decision 


The  quality  of  being  trusted  and  believed  in  or  of  being  convincing  or 
believable  (based  on  The  Oxford  English  Dictionary). 

The  individual  who  pays  the  bill  for  the  study. 

Facts  or  other  pieces  of  qualitative  and/or  quantitative  information. 

The  result  of  making  up  one’s  mind  regarding  a  choice  between 
alternatives. 


Decision  maker  An  individual  who  makes  up  his/her  mind  regarding  the  problematic 

situation  or  concerning  the  recommendations  by  (outcomes  of)  the 
study  in  preparation  of  decisions  by  a  higher  authority. 

Delphi  method  A  structured  communication  technique,  originally  developed  as  a 

systematic,  interactive  forecasting  method  which  relies  on  a  panel  of 
experts  (http://en.wikipedia.org/wiki/Delphi_method;  accessed  on 
18  October  2011). 

Due  diligence  An  investigation  based  on  agreed  standards  of  the  (true)  worth  of 

something. 
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End  user 

An  individual  who  is  ultimately  affected  (either  positively  or 
negatively)  by  the  study  outcomes  and  related  decisions. 

Expert  (subject-matter  expert) 

An  individual  who  has  considerable  relevant  knowledge  in  a  particular 
area  without  necessarily  owning  or  otherwise  being  part  of  the 
problematic  situation. 

Facilitated  modelling 

A  process  where  the  analyst  actively  engages  the  client  (group)  to 
participate  in  all  stages  of  a  ‘soft’  OA  study  and  thus  acts  as  an 
facilitating  analyst  (often  with  the  help  of  a  supporting  facilitator). 

Facilitator 

An  individual  who  helps  (i.e.  supports,  enables  and  encourages) 
individuals  or  a  group  or  groups  of  stakeholders,  clients,  subject  matter 
experts,  and  analysts  work  together  through  the  stages  of  a  study  by 
managing  procedure  (the  way  the  problem  is  tackled)  and  process  (the 
way  participants  interact),  whilst  adopting  an  impartial  attitude. 

‘Flard’  Operational  Analysis 

OA  based  on  well  understood,  standard  science  and  mathematics. 

Input 

Data  needed  for  setting  up  or  running  a  model  or  influencing  a  process 
with  the  aim  of  affecting  the  results  of  that  model  or  process. 

Judgement 

The  result  of  the  process  of  forming  an  opinion. 

MA 

Morphological  Analysis 

MC(D)A 

Multiple  Criteria  (Decision)  Analysis 

Mess 

Any  problematic  situation  that  requires  or  seems  to  require 
(re)solutions  involving  some  action,  although  it  is  unclear  initially 
what  the  issues  are,  what  actions  are  required  and/or  who  is  or  should 
be  involved. 

Method 

A  structured  set  of  guidelines  or  rules  to  achieve  some  clear  well- 
defined  purpose. 

Methodology 

A  particular  combination  of  methods  that  possess  a  common  set  of 
characteristics  and  assumptions  and  are  used  in  a  study  to  achieve  a 
purpose. 

Model 

A  representation  of  (a  part  of)  reality  (i.e.  a  problematic  situation)  as 
seen  by  a  number  of  people  who  wish  to  use  the  model  to  understand, 
manage,  or  improve  that  reality. 

Morphological  Analysis 

A  method  for  exploring  all  the  possible  solutions  to  a  multi¬ 
dimensional,  non-quantified  problem  complex  (http://en.wikipedia.org/ 
wiki/Morpho logical  analysis,  accessed  on  17  October  201 1). 

Multi -methodology 

A  combination  of  methods  from  possibly  different  methodologies. 

Multiple  Criteria  (Decision)  Analysis 

A  sub-discipline  of  OA  that  explicitly  considers  multiple  criteria  in 
decision-making  environments  (http://en.wikipedia.org/wiki/Multi- 
criteria  decision  analysis,  accessed  on  17  October  201 1). 
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NATO 

OA 

Objective 


Operational  Analysis 

Option 

OR 

Outcome 

Primary  client 

Problem 

Problem  (proper) 

Problem  structuring 
Problem  Structuring  Methods 

Problematic  situation 

Puzzle 

Qualitative 


ORGANIZATION 


North-Atlantic  Treaty  Organisation 
Operational  Analysis 

(adjective  in  connection  with  data)  Based  on  agreed,  established, 
consensually  observed  facts. 

(adjective  in  connection  with  an  activity  or  process)  Based  on  rules 
grounded  in  theory  or  established  practice  and  characterised  by 
recorded  argumentation  and  rationale  and  following  an  agreed  and 
sound  process  accepted  by  all  involved; 

(noun)  Aim,  purpose  or  goal. 

The  interdisciplinary  science  that  focuses  on  how  appropriate  actions 
can  be  designed  to  change  (i.e.  towards  improvement)  or  even 
(re)solve  problematic  situations. 

One  of  two  or  more  possibilities  that  can  be  chosen  for  (re)solving  a 
problematic  situation. 

Operational  Research  (equal  to  Operational  Analysis) 

This  denotes  what  has  been  achieved  by  the  study  (‘effect’  or 
‘impact’),  based  on  specific  results  and  their  analytic  interpretation. 

The  term  ‘primary  client’  refers  to  the  single  individual  or  (small) 
group  of  individuals  who  belong  to  the  client  system  and  who  acts  or 
are  acting  as  a  partner  (first  point  of  contact)  to  the  analyst  throughout 
the  conduct  of  the  study. 

Any  situation  where  there  is  a  perceived  gap  between  its  current  state 
and  its  desired  or  required  state,  which  needs  or  seems  to  need  a 
(re)solution  involving  some  action,  although  it  may  often  be  unclear 
what  (re)solution  and  what  action  are  most  appropriate. 

A  problematic  situation  in  which  it  is  not  very  clear  initially  what  the 
issue  is  and  how  a  solution  can  be  found;  could  be  positioned  between 
‘Puzzle’  and  ‘Mess’  (ref.  Chapter  2). 

The  process  of  analysing  a  problematic  situation  in  order  to  identify 
the  issue(s)  to  be  resolved. 

Methods  developed  for  dealing  with  unstructured  problematic 
situations  that  are  characterised  by  multiple  actors,  their  multiple 
perspectives  and  conflicts  of  interest,  uncertain  and  unquantifiable 
factors,  and  designed  to  support  groups  in  their  decision  making. 

Any  situation  that  is  characterised  by  the  existence  of  one  or  more 
problems  and  requires  or  seems  to  require  (re)solutions  involving  some 
action,  although  it  may  often  be  unclear  initially  what  action. 

A  problematic  situation  in  which  it  is  fairly  clear  what  the  issue  is  and 
how  a  solution  can  be  found. 

Based  on  distinctions  or  descriptions  in  terms  of  characteristics, 
gradation  or  order. 
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Quantitative 


Based  on  measurement  in  tenns  of  numbers  expressed  on  a  numerical 
scale. 


Repeatability 


Requisite 


Result 


Rigour 


The  quality  of  a  phenomenon  to  occur  again,  possibly  in  different 
places  and  times,  and  observed  by  different  people. 

The  characteristic  of  something  when  relevant  people  have  agreed  on 
its  adequateness  and  sufficiency  for  use. 

This  denotes  what  has  been  produced  by  applying  a  model,  method  or 
methodology,  or  merely  following  a  process. 

Quality  achieved  through  strict  enforcement  of  logical  rales  and 
doctrine. 


Robustness 

SAS 

Satisficing 

Scrutineer 

SME 

SODA 


The  quality  of  withstanding  or  overcoming  adverse  conditions  (based 
on  The  Oxford  English  Dictionary). 

System  Analysis  and  Studies 

A  decision  or  strategy  or  (re)solution  is  satisficing  if  it  is  good  enough 
(‘adequate’),  but  not  necessarily  optimal. 

An  individual  who  provides  an  independent  review  of  the  study. 

Subject-Matter  Expert  (see  under  Expert) 

Strategic  Options  Development  and  Analysis 


‘  Soft’  Operational  Analysis  Operational  Analysis  exploiting  methodologies  and  methods  that  are 

predominantly  based  on  the  rational  (i.e.  not  intuitive)  use  of  human 
judgement. 

Soft  Systems  Methodology  An  approach  using  ‘real  world’  descriptions  and  systems  engineering 

terms  to  structure  messy  problems  where  there  may  be  divergent  views 
about  the  nature  and  definition  of  the  problem. 

Sponsor  The  individual  who  actually  owns  the  study 

SSM  Soft  Systems  Methodology 


Stakeholder 


An  individual  who  can  affect  or  can  be  affected  by  the  (resolution  of 
the)  problematic  situation. 


Strategic  Options  Development  and  Analysis 

An  approach  using  diagram-based  visualisations  to  explore  messy 
problems  and  capture  individual  and  group  views  with  the  aim  of 
reaching  a  shared  understanding  of  the  problem. 

Study  phase  A  coherent  and  distinguishable  part  of  a  sequence  or  cycle  of  activities 

over  time. 


Subjective  Based  on  personal  feelings  or  intuition  or  expertise. 

System  A  collection  of  organised  things  constituting  a  whole  and  the 

relationships  between  its  constituent  parts. 
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Technique 


Denotes  each  of  the  specific  algorithms  or  interviewing  modes  (etc.) 
which  are  part  of  a  method. 


TG 


Task  Group 


Tool 

Transparency 

Triangulation 

Uncertainty 

Validation 

Validity 


Either  a  ‘Technique’  or  the  agent  of  a  technique  (e.g.  a  software 
program). 

The  quality  of  being  easy  to  perceive  or  detect,  or  open  to  public 
scrutiny  (based  on  The  Oxford  English  Dictionary). 

The  use  and  critical  synthesis  of  different  sources  or  perspectives  or 
methods  regarding  a  (usually)  single  issue. 

The  absence  of  knowledge  and  the  inherent  variability  of  phenomena, 
both  in  varying  degrees. 

The  activity  of  proving  or  demonstrating  or  supporting  the  truth  or 
accuracy  or  value  of  something  (based  on  The  Oxford  English 
Dictionary). 

The  quality  of  being  logically  or  factually  sound  (‘cogent’)  (based  on 
The  Oxford  English  Dictionary);  fitness  for  purpose. 


Wicked  problem  A  problem  that  is  difficult  or  impossible  to  solve  because  of 

incomplete,  contradictory,  and  changing  requirements  that  are  often 
difficult  to  recognise,  (http://en.wikipedia.org/wiki/wicked_problem; 
accessed  on  2  November  2011). 

See  ‘Mess’;  this  CoBP  uses  ‘mess’  rather  than  ‘wicked  problem’. 
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Preface 


ORGANIZATION 


The  Task  Group  (TG-034,  SAS  Activity  087)  was  commissioned  by  the  System  Analysis  and  Studies  (SAS) 
Panel  to  produce  a  code  of  best  practice  for  judgement-based  Operational  Analysis.  After  an  exploratory  phase 
in  2009  which  addressed,  at  the  Panel’s  request,  the  issues  surrounding  use  of  judgement  in  Operational 
Analysis  (OA),  the  TG  formally  started  its  work  Spring  20 1 0  and  finalised  the  code  at  the  end  of  20 1 1 . 

The  TG  began  by  identifying  a  number  of  topics  that  would  have  to  be  addressed  and  considering  what  the 
specific  boundaries  of  a  code  of  best  practice  would  be.  This  included  its  readership.  Inspired  and  informed  by 
discussions  at  two  academic  conferences1  about  a  number  of  challenging  propositions  regarding  the  subject 
proposed  by  the  TG,  the  TG  went  on  to  identify  and  (re-)read  relevant  literature. 

The  TG  decided  to  write  a  Guide  consisting  of  three  volumes:  an  analyst-oriented  document  (the  “Code  of  Best 
Practice  for  ‘Soft’  Operational  Analysis”),  a  client-oriented  document,  and  a  brief  summarising  brochure  for 
high-level,  ‘executive’  decision  makers.  The  TG  also  decided  to  write  the  analyst-oriented  document  first, 
not  only  in  view  of  the  crucial  role  of  the  analyst,  but  also  because  the  TG  Members  considering  their  background 
generally  felt  more  comfortable  with  an  analyst’s  perspective.  The  client-oriented  volume  was  developed  after 
a  first  full  draft  of  the  analyst-oriented  volume  and  saw  an  addition  of  specific  topics  relevant  to  clients  keeping 
mutual  consistency  in  mind.  The  summary  brochure  was  derived  from  the  client-oriented  volume. 

The  design  and  writing  of  the  Guide’s  volumes  was  assigned  individually  to  the  following  participating 
nations/agencies:  NATO/ACT,  Australia,  Canada,  Germany,  Netherlands,  Sweden,  and  the  United  Kingdom. 
The  final  set  of  volumes  was  submitted  to  the  Panel  in  December  2011.  Their  common  main  title  is  “NATO 
Guide  for  Judgement-Based  Operational  Analysis  in  Defence  Decision  Making”,  referred  to  as  ‘the  Guide’. 
The  analyst-oriented  volume  however  is  the  Code  of  Best  Practice  (CoBP)  proper.  The  other  two  volumes  have 
a  more  explanatory  nature:  one  is  directed  to  clients  and  is  organised  around  seven  key  questions  that  clients  are 
likely  to  ask,  the  other  is  a  short  summary  of  a  few  key  aspects  of  (using)  judgement-based  OA  and  is  written 
for  executive  decision  makers  explaining  and  promoting  key  aspects. 

The  TG  used  existing  publications,  including  textbooks,  edited  collections  (proceedings)  and  reviewed  papers, 
in  order  to  attempt  to  produce  useful  guidance,  based  on  each  TG  Member’s  personal  knowledge  and 
experience.  The  TG  did  not  always  try  to  reconcile  differing  academic  views  but  rather  made  an  effort  to 
synthesise  all  useful  ingredients  where  possible  and  made  informed  choices  where  a  synthesis  appeared  to  be 
infeasible.  This  has  all  been  done  with  the  Guide’s  general  and  the  CoBP’s  specific  purpose  in  mind  and 
creating,  along  the  way,  clarity  and  focus  in  the  vast  and  rapidly  further  developing  world  of  ‘soft’  OA. 

In  order  to  illustrate  issues,  the  TG  used  a  number  of  unclassified  descriptions  of  case  studies  and  derived  short 
pieces  of  text  from  them  each  addressing  a  particular  issue.  They  appear  as  illustrative  text  boxes  in  the  main 
text  of  this  volume.  The  TG  decided  on  which  issues  would  require  clarification  by  text  boxes  and  which 
aspects  from  the  available  case  summaries  would  be  appropriate.  In  addition,  there  are  some  clarifying  text 
boxes  that  are  not  based  on  cases  and  text  boxes  at  the  start  of  most  chapters,  summarising  their  main  points 
both  as  statements  and  as  recommendations  to  the  analyst.  Different  colouring  is  used  to  denote  the  three 
different  text  box  types. 

The  TG  has  restricted  itself  in  referencing  the  material  in  the  main  text  in  order  to  not  distract  the  reader  too 
much  from  the  content.  By  its  nature,  the  CoBP  (in  fact  the  entire  Guide)  is  a  work  of  review  and  representation 


ISMOR  (30  August  -  3  September  2010,  New  Place,  Hampshire,  UK):  27th  International  Symposium  on  Military  Operational 
Research,  with  special  theme  ‘The  use  of  ‘soft’  methods  in  OR’. 

OR52  (7-9  September  2010,  Royal  Holloway  University  of  London,  Egham,  UK):  52nd  Annual  International  Conference  on 
Operational  Research  of  the  O.R.  Society. 
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of  relevant  ideas.  If  a  particular  method  for  approaching  a  problem  or  issue  was  identified  in  an  academic  paper 
or  book  and  adapted  for  use  in  the  CoBP,  a  reference  to  the  source  is  given.  So,  references  are  given  where 
appropriate  and  are  not  designed  to  be  exhaustive,  or  even  comprehensive.  Some  chapters  contain  a  short  list  of 
recommended  publications  for  further  reading.  The  references  and  the  lists  of  publications  for  further  reading 
encompass  the  literature  that  the  TG  have  used  for  the  CoBP  and  serve  as  a  tribute  to  our  major  sources. 

The  TG  very  much  appreciates  the  efforts  made  by  external  reviewers  to  review  the  documents.  A  draft  version 
of  the  analyst-oriented  volume  was  reviewed  by  two  associate  professors  ,  amended  based  on  this  review  and 
then  finalised.  A  draft  of  the  client-oriented  volume  was  reviewed  by  three  individuals3  who  belong  to  the 
Defence  community,  and  finalised  using  their  commentary.  The  TG  acknowledges  this  support  and  thanks  all 
reviewers  for  their  most  valuable  advice.  The  Chair  would  like  to  thank  all  contributing  nations  and  agencies 
for  their  effort  in  writing  these  documents  which  will  be  of  benefit  to  all  NATO  Nations,  PfP  and  Contact 
Nations,  other  Nations  and  organisations,  and  individuals. 

Although  the  authors  expect  that  the  Guide  will  be  useful  as  it  is  written,  this  work  will  clearly  gain  value  from 
practical  experience  in  working  with  it.  The  TG  therefore  recommends  to  the  Panel  that,  in  due  course,  an  activity 
should  be  set  up  to  see  if  the  Guide,  and  especially  the  CoBP,  needs  adjustment  based  on  the  feedback  that, 
hopefully,  would  have  been  brought  to  the  Panel’s  or  the  authors’  attention.  That  would  be  good  practice. 


Dr.  L.A.  Franco  (University  of  Hull,  United  Kingdom);  Dr.  E.A.J.A.  Rouwette  (Radboud  University  of  Nijmegen,  Netherlands). 

Dr.  R.A  Forder  (fomierly  at  Defence  Science  &  Technology  Laboratory,  United  Kingdom);  Lt.Col.  J-H.  Pay  (Norwegian 
Defence  Research  Establishment  FFI,  Norway);  Cpt.Cdr.  F.S.  Ordean  and  Lt.Col.  I.  Psomas  (Joint  Assessment  Branch,  Joint 
Force  Command  Brunssum,  NATO). 


RTO-TR-SAS-087 


xv 


SAS-087  Task  Group  Membership 


ORGANIZATION 


AUSTRALIA 

N . J.  Curtis 

Research  Leader  Operations  Research 

Land  Operations  Division 

Defence  Science  &  Technology  Organisation 

P.O.Box  1500 

Edinburgh,  SA  5 1 1 1 

Email:  neville.curtis@dsto.defence.gov.au 

CANADA 
M.  Halbrohr 

Director  Strategic  Military  Personnel  Research 
and  Analysis,  National  Defence  Headquarters 
Major-General  George  R.  Pearkes  Building 
101  Colonel  By  Drive 
Ottawa,  Ontario  K1A  0K2 
Email:  mira.halbrohr@forces.gc.ca 

FRANCE 

O.  de  Baysier  (until  Autumn  2010) 
DGA/DS/CATOD 

16bis,  Avenue  Prieur  de  la  Cote  d’Or 
94114  Arcueil  Cedex 

GERMANY 

G.  Mihelcic 

Chair  of  Operations  Research 
Dept,  of  Computer  Science  /  INF1 
COMTES  SA 

Bundeswehr  University  Munich 
Werner-Heisenberg-Weg  39 
85577  Neubiberg 
Email:  goran.mihelcic@unibw.de 

Lt.Col.  C.  Wittmann 
Division  III  (OR  Branch) 

Bundeswehr  Transformation  Centre 
Einsteinstr.  20 
85521  Ottobrunn 

Email:  christianwittmann@bundeswehr.org 

NETHERLANDS 

I.C.L.  Bastings 

TNO,  Behavioural  and  Societal  Sciences 
Dept,  of  Military  Operations 

P. O.  Box  9684 
2509  JG  The  Hague 

Email:  ingrid.bastings@tno.nl 


D.J.D.  Wijnmalen  (Chair) 

TNO 

Behavioural  and  Societal  Sciences 
Dept,  of  Networked  Organisations 
P.O.  Box  9684 
2509  JG  The  Hague 
Email:  diederik.wijnmalen@tno.nl 

SWEDEN 
J.  Frelin 

FOI,  Dept,  of  Defence  Analysis 
S- 16490  Stockholm 
Email:  jan.frelin@foi.se 

UNITED  KINGDOM 

S.M.  Lord 

Policy  and  Capabilities  Studies  Department 
Defence  Science  &  Technology  Laboratory 
Level  C-J  West  Court 
Portsdown  West,  Portsdown  Hill  Road 
Fareham,  Hampshire  POl  76AD 
Email:  smlord@dstl.gov.uk 

G.A.  Pickbum 

Defence  Science  &  Technology  Laboratory 
Level  C-J  West  Court 
Portsdown  West,  Portsdown  Hill  Road 
Fareham,  Hampshire  POl 7  6AD 
Email:  gapickbum@dstl.gov.ulc 

UNITED  STATES 

Y.H.  Wong  (as  from  December  2010) 
Operations  Analysis  Division  (OAD) 

Marine  Corps  Combat  Development  Command 
(MCCDC) 

3300  Russell  Road 
Quantico,  VA 

Email:  yuna.wong@usmc.mil 

NATO/ACT 

S.  Collins 

Operational  Analysis 
Capability  Engineering  Division 
NATO  HQ  Supreme  Allied  Commander 
Transformation  (SACT) 

7857  Blandy  Road,  Suite  100 
Norfolk,  VA  23551-2490 
United  States 

Email:  sue.collins@act.nato.int 


XVI 


RTO-TR-SAS-087 


NATO/ACT  (cont’d) 

A.  Smethurst 
Operational  Analysis 
Capability  Engineering  Division 
NATO  HQ  Supreme  Allied  Commander 
Transformation  (SACT) 

7857  Blandy  Road,  Suite  100 
Norfolk,  VA  23551-2490 
United  States 

Email :  alex. smethurst@act. nato . int 


RTO-TR-SAS-087 


XVII 


xviii 


RTO-TR-SAS-087 


NATO  Guide  for  Judgement-Based  Operational 
Analysis  in  Defence  Decision  Making 
(RT  O-TR-S  AS-087) 

Executive  Summary 

Judgement  in  different  guises  has  been  used  by  military  staff  whenever  assessing  problematic  situations 
and  making  decisions.  NATO  practitioners  have  determined  that  approaches  (i.e.  theories,  methods, 
techniques,  models)  within  Operational  Analysis  (OA)  that  are  predominantly  based  on  human  judgement, 
are  an  increasingly  critical  capability  needed  to  support  defence  decision  making. 

The  field  of  judgement-based  methodologies  and  methods  within  OA  is  usually  known  as  ‘soft’  OA.  It  has 
a  significantly  more  qualitative  and  subjective  nature  than  the  traditional,  ‘hard’,  OA  methodologies  which 
tend  to  be  more  quantitative  and  objective.  Wider  acceptance  and  employment  of  judgement-based  OA 
would  enable  the  defence  community  to  enhance  its  capability  to  deal  with  complex,  high  priority  problem 
issues  for  which  this  type  of  analysis  is  particularly  suited. 

The  development  of  the  “NATO  Guide  for  Judgement-based  Operational  Analysis  in  Defence  Decision 
Making”  was  commissioned  by  the  System  Analysis  and  Studies  Panel  within  NATO’s  Research  and 
Technology  Organisation.  It  is  intended  to  increase  the  acceptance  of  judgement-based  OA  within  the 
military  and  defence-oriented  operational  analyst  communities.  It  provides  pragmatic  guidelines  to  enable 
this  type  of  OA  to  be  practised  in  a  more  rigorous  and  scientific  manner,  especially  by  those  who  are  new 
to  this  type  of  approach.  The  Guide  is  also  designed  to  be  informative  in  the  sense  that  it  aims  to  clarify 
what  judgement-based  OA  is,  and  when  and  how  it  can  be  used. 

The  purpose  of  the  Guide  is  therefore: 

•  To  create  an  understanding  of  what  judgement-based  OA  is; 

•  To  clarify  what  judgement-based  OA  can  do  to  help  address  problematic  situations,  and  what  people 
can  expect  from  it  in  that  respect;  and 

•  To  provide  guidance  on  how  a  judgement-based  OA  study  should  be  carried  out  to  maximise  the 
validity,  credibility  and  acceptance  of  the  study  and  its  outcomes. 

The  Guide  is  published  as  three  volumes:  an  analyst-oriented  document  (the  “Code  of  Best  Practice  for 
‘Soft’  Operational  Analysis”),  a  client-oriented  document,  and  a  brief  summarising  brochure  for  high-level, 
‘executive’  decision  makers  explaining  key  aspects. 

This  volume  of  the  Guide  sets  the  rules  of  the  road  for  analysts  who  are  new  to  this  field  of  OA.  It  is 
expected  that  more  experienced  ‘soft’  OA  analysts  will  also  find  it  of  value  for  educational  or  reference 
purposes  whilst  having  developed  their  own  views  on  how  these  kind  of  analyses  should  be  carried  out 
and  perhaps  even  their  own  ‘soft’  methods  or  methodologies.  The  guidelines  are  offered  as  a  means  of 
assistance  to  analysts  and  are  not  meant  to  be  too  prescriptive  in  nature.  They  address: 

•  The  nature  of  ‘soft’  OA  and  the  type  of  problematic  situations  which  require  this  type  of  analysis; 

•  How  to  achieve  validity,  credibility  and  acceptance  of  ‘soft’  OA  studies; 

•  The  roles  and  responsibilities  of  individuals  involved  in  a  ‘soft’  OA  study; 
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•  The  design  of  a  study,  the  design  of  the  study’s  (multi-)methodology  and  how  to  recognise  and 
cope  with  problematic  situations; 

•  How  to  deal  with  data,  subject  matter  experts,  bias  and  aspects  related  to  workshops;  and 

•  How  to  communicate  the  results  with  the  client. 

The  other  two  volumes  explain  to  clients  what  judgement-based  OA  is,  in  what  cases  it  may  be  useful  to 
them,  how  their  acceptance  of  a  study’s  outcomes  can  be  enhanced,  and,  when  they  are  involved  in  a  study, 
what  their  roles  and  responsibilities  should  be. 
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Guide  OTAN  pour  l’analyse  operationnelle  basee  sur 
le  jugement  dans  la  prise  de  decision  de  defense 

(RT  O-TR-S  AS-087) 


Synthese 

Les  etats-majors  ont  utilise  le  jugement  sous  differents  aspects,  a  chaque  fois  qu’ils  ont  evalue  des  situations 
problematiques  et  pris  des  decisions.  Les  professionnels  de  l’OTAN  ont  etabli  que  les  approches  utilisees 
(c.a.d.  les  theories,  methodes,  techniques  et  modeles)  lors  de  l’analyse  operationnelle  (AO),  qui  sont 
principalement  basees  sur  le  jugement  humain,  represented  une  capacite  critique  croissante  indispensable  au 
soutien  des  prises  de  decision  de  defense. 

La  partie  de  l’AO  dont  la  methodologie  et  les  methodes  sont  basees  sur  le  jugement  est  connue  comme  AO 
«  en  douceur  ».  Elle  est,  de  maniere  significative  et  par  nature,  plus  qualitative  et  subjective  que  LAO 
classique  («  hard  »)  dont  la  methodologie  est  davantage  quantitative  et  objective.  Une  meilleure  acceptation 
et  une  meilleure  utilisation  de  l’AO  permettraient  a  la  communaute  de  la  Defense  d’ameliorer  sa  capacite  a 
traiter  des  problemes  complexes  et  hautement  prioritaires  pour  lesquels  ce  type  d’analyse  conviendrait 
particulierement. 

L’elaboration  du  «  Guide  OTAN  pour  Tanalyse  operationnelle  basee  sur  le  jugement  dans  la  prise  de 
decision  de  defense  »  a  ete  soutenue  par  la  Commission  sur  les  etudes  et  Tanalyse  de  systemes  de 
TOrganisation  pour  la  Recherche  et  la  Technologie  de  l’OTAN.  II  est  congu  pour  accroitre  Tapprobation 
des  analyses  operationnelles  (AO)  basee  sur  le  jugement  au  sein  des  communautes  militaires  et  d’analyse 
operationnelle  axes  sur  la  defense.  II  foumit  des  principes  directeurs  pragmatiques  pour  que  ce  type  d’AO 
soit  realise  de  maniere  plus  rigoureuse  et  scientifique,  particulierement  par  ceux  qui  decouvrent  ce  type 
d’approche.  Ce  guide  est  egalement  a  caractere  informatif,  car  il  vise  a  clarifier  ce  qu’est  TAO  basee  sur  le 
jugement  et  ou  et  quand  en  faire  usage. 

L’objectif  de  ce  guide  est  de  : 

•  Sensibiliser  a  TAO  basee  sur  le  jugement ; 

•  Clarifier  la  maniere  dont  TAO  basee  sur  le  jugement  peut  aider  a  traiter  des  situations  problematiques, 
et  ce  que  les  personnes  peuvent  en  attendre  ;  et 

•  Foumir  des  orientations  sur  la  maniere  dont  une  etude  par  AO  basee  sur  le  jugement  doit  etre  menee 
afin  d’optimiser  la  validite,  la  credibility  et  Tacceptation  de  ladite  etude  et  de  ses  resultats. 

Le  guide  est  constitue  de  trois  ouvrages  :  un  document  d’analyste  (le  «  Code  de  bonnes  pratiques  pour  une 
analyse  operationnelle  ‘en  douceur’  »)  definissant  les  regies  a  suivre  par  les  analystes,  un  document  oriente 
«  client »,  ainsi  qu’un  court  resume  sous  forme  de  brochure  a  T attention  des  decideurs  (de  haut  niveau, 
de  «  Texecutif  »)  expliquant  les  principaux  aspects  de  cette  analyse. 

Le  present  volume  de  ce  guide  etablit  les  regies  a  suivre  pour  les  analystes  qui  decouvrent  ce  champ  d’AO. 
II  est  fort  probable  que  les  analystes  d’AO  fonctionnelle  plus  experimentes  le  trouvent  egalement  utile  a 
des  fins  d’enseignement  et  de  reference,  bien  qu’ils  aient  developpe  leurs  propres  opinions  sur  T  execution 
de  ce  genre  d’analyse  et  peut-etre  meme  developpe  leurs  propres  methodes  ou  methodologies 
fonctionnelles.  Les  principes  directeurs  sont  proposes  en  guise  d’aide  aux  analystes  et  ne  sont  pas  prevus 
pour  etre  trop  normatifs  de  nature.  Ils  concement : 
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•  La  nature  des  AO  fonctionnelles  et  le  type  de  situations  problematiques  qui  necessitent  ce  type 
d’analyse  ; 

•  La  maniere  d’etablir  la  validite,  asseoir  la  credibility  et  obtenir  l’acceptation  des  etudes  d’AO 
fonctionnelle ; 

•  Les  roles  et  responsabilites  des  personnes  engagees  dans  une  etude  d’AO  fonctionnelle  ; 

•  La  conception  d’une  etude,  la  conception  de  la  methodologie  (multiple)  d’une  etude  et  la  maniere 
de  reconnaitre  et  d’affronter  des  situations  problematiques  ; 

•  La  maniere  de  traiter  les  donnees,  les  experts  en  la  matiere,  la  partialite  et  les  aspects  lies  aux 
ateliers  ;  et 

•  La  maniere  de  communiquer  les  resultats  au  client. 

Les  deux  autres  volumes  expliquent  aux  clients  ce  qu’est  LAO  basee  sur  le  jugement,  dans  quels  cas  elle 
peut  leur  etre  utile,  comment  ameliorer  l’acceptation  des  resultats  d’une  etude,  et,  lorsque  les  clients  sont 
engages  dans  une  etude,  quels  devraient  etre  leurs  roles  et  responsabilites. 
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Chapter  1  -  INTRODUCTION 


ORGANIZATION 


1.1  BACKGROUND 

The  defence  environment  is  complex  and  dynamic.  This  is  a  result  of,  the  nature  of  modem  conflict,  rapid 
changes  in  technology,  the  need  to  deal  with  uncertainty  in  the  face  of  limited  resources,  changed  attitudes  to 
risk,  and  the  sheer  diversity  of  actors  from  different  cultural  backgrounds.  Defence  decision  makers  are 
confronted  with  an  increasing  operational  complexity  which  has  strategic  implications.  Decisions  on  defence 
policy  and  strategy  are  characterised  by  uncertainty  and  risk.  Planning,  conducting  and  evaluating  missions 
also  include  indirect  and  non-kinetic  effects,  dealing  with  other  (non-military)  actors,  social  effects  on  the 
local  population,  public  sentiment  and  other  human  factors.  In  addition,  defence  decision  makers  may  be 
involved  at  any  stage  of  the  defence  material  cycle  ranging  from  conceptual  development,  through  to 
acquisition  to  use  in  conflict.  Similarly,  decisions  need  to  be  made  in  personnel-based  activities  such  as 
career  structures  and  training  regimes. 

All  these  decisions  are  made  at  strategic,  operational  or  tactical  levels.  They  may  be  aimed  at  the  present 
or  the  far  future,  and  may  involve  choices  among  tangible  (materiel)  and/or  intangible  (organisational 
structure,  strategies,  etc.)  alternatives.  Critically,  many  decisions  are  required  where  judgement  rather  than 
known  facts  plays  a  key  role. 

NATO  practitioners  have  determined  (e.g.  [1]) 
that  approaches  (i.e.  theories,  methods, 
techniques,  models)  within  Operational 
Analysis  (OA)  that  are  predominantly  based  on 
human  judgement,  are  an  increasingly  critical 
capability  needed  to  support  defence  decision 
making.  Judgement  in  different  guises  has 
been  used  by  military  staff  whenever  assessing 
problematic  situations  and  making  decisions. 

The  field  of  judgement-based  methodologies 
and  methods  within  OA  is  usually  known  as 
‘soft’  OA.  It  has  a  significantly  more 
qualitative  and  subjective  nature  than  the 
traditional,  ‘hard’,  OA  methodologies  which  tend 

Military  ‘hard’  OA  has  a  long  history.  During  World  War  I  both  the  American  and  British  governments 
established  boards  to  support  defence  decision-making  issues  with  mathematical  methods  [2].  (Military) 
OA  as  a  formal  discipline  started  to  be  recognised  in  the  late  1930s  and  subsequent  years  by  contributing 
to  addressing,  for  example,  air  defence,  bombing  tactics,  submarine/convoy  war  and  logistic  problems 
during  World  War  II. 

‘Hard’  OA  predominantly  relies  on  mathematically  expressed  relationships  between  variables  derived 
from  physical  theory  using  quantitative  data  and  seeks  to  identify  quantitative  (possibly  optimal)  solutions 
to  problems,  given  a  set  of  (well-defined)  restrictions.  ‘Hard’  OA  consists  of  traditional  OA  approaches 
such  as  linear  programming,  queuing  theory,  computer  simulation,  etc.,  which  are  all  characterised  by 
their  well-structured  engineering-like  nature  and  their  reliance  on  mathematically  expressed  relationships 
between  variables. 

‘Soft’  OA  started  to  be  recognised  in  the  early  1960s  when  the  Rand  Corporation  first  reported  on  the 
Delphi  technique  as  an  approach  to  achieving  consensus  among  experts  regarding  bombing  requirements 
[3].  In  the  late  1960s  Churchman  wrote  an  editorial  [4]  drawing  attention  to  a  type  of  (‘wicked’)  problem 


In  support  of  Australia’s  deployment  to  East  Timor,  a  study 
was  made  of  the  triggers  and  causes  that  have  led  to 
disruptive  events  in  that  country’s  400  year  history. 
Importantly,  the  analyst  was  able  to  present  the  study  in 
terms  that  could  be  readily  used  to  forecast  possible  events 
that  could  occur  as  the  population  reacted  to  the  changed 
socio-political  environment.  This  enabled  ‘what-if  studies 
and  campaign  metrics  to  be  devised. 

This  type  of  study  should  be  treated  with  confidence  as  in 
such  cases  the  skill  lies  in  the  analysis  not  necessarily  the 
domain  area,  as  soft  OA  is  applicable  to  many  areas.  The 
client  should  therefore  exploit  the  fact  that  a  military  ‘soft’ 
operational  analyst  is  able  to  provide  useful  insight  into 
areas  well  outside  of  standard  conflict  analysis. 


to  be  significantly  more  quantitative  and  objective. 
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that  was  different  from  the  problems  usually  addressed  by  OA  [5].  Since  the  1970s  the  UK  originated  a 
number  of  OA  methods  specifically  designed  for  dealing  with  this  type  of  problems;  they  are  still  in  use 
today  [6]. 

The  strength  of  ‘soft’  OA  lies  in  its  ability  to  address  complex  (i.e.  confusingly  unstructured)  situations  in 
which  what  needs  to  be  done  and  how  to  do  it  are  both  problematic,  and  to  deal  with  the  critical  human 
aspects  of  the  situation  (e.g.  differences  in  stakeholder  perspectives  and  agendas,  organisational  politics, 
disagreement  on  ways  forward,  etc.)-  It  is  characterised  by  its  use  of  human  judgement  in  the  form  of 
(informed)  opinions  from  experts  and  other  stakeholders,  thereby  allowing  more  degrees  of  freedom  and 
relying  less  on  mathematical  modelling  in  comparison  with  ‘hard’  OA. 

The  discussion  of  the  limits  of  a  purely 
mathematical,  formal  approach  to  OA  is  not 
new.  In  fact,  even  from  within  the  ‘hard’ 

Operational  Research  (OR)  community  there 
have  been  critiques  and  examples  of  dissenters 
(e.g.  [7]):  one  is  faced  with  ill-structured, 

‘messy’  situations  in  which  decisions  have  to 
be  made;  yet  one  does  not  completely  comprehend  the  situation.  Here,  ‘soft’  OA  claims  to  help  the  decision 
maker  either  as  an  individual  or  within  a  group. 

Moreover,  a  problem  definition  does  not 
present  itself  automatically.  Before  any  type  of 
model  can  be  developed,  the  characteristics 
and  boundaries  of  a  problem,  the  objective(s), 
the  multiple  factors  and  their  (critical) 
relationships  and  dependencies  that  are  of 
relevance  need  to  be  identified  and  structured. 

And,  at  the  end  of  even  a  ‘hard’  analysis,  issues  related  to  the  interpretation  of  results  and  feedback  often 
still  involve  judgement.  Operational  analysts  therefore  need  to  be  familiar  with  both  ‘hard’  and  ‘soft’  OA 
if  they  are  to  be  successful  in  addressing  the  diversity  of  military  client  tasking.  Practice  has  shown  that 
‘soft’  and  ‘hard’  OA  can  usefully  complement  each  other.  They  need  not  be  competing  approaches. 

The  application  of  ‘soft’  OA  on  defence  problems  is  manifold  and  often  successful,  but  success  is  not 
guaranteed.  Although  ‘soft’  OA  has  been  accepted  and  successfully  used  by  some  analysts  within  NATO’s 
defence  communities  for  structuring  problems,  option  design,  option  evaluation  and  action  planning,  it  is 
perceived  that  there  is  still  reluctance  in  some  quarters  to  employ  such  methods.  This  lack  of  acceptance 
and  use  may  be  attributed  to  several  reasons,  including:  a  lack  of  affinity  to  using  such  methods  by 
analysts  who  have  been  trained  in  ‘hard’  disciplines  such  as  engineering  and  mathematics;  philosophical 
disputes  as  to  whether  ‘soft’  OA  truly  is  OA  given  its  predominantly  qualitative  or  semi-quantitative 
nature;  and  questions  about  its  scientific  rigour  and  quality  control  measures. 

Progress  has  been  made  in  laying  the  theoretical  foundations  for  the  rigorous  use  of  ‘soft’  OA  both  in 
isolation  and  as  part  of  a  multi-methodological  systems-type  OA  approach.  In  particular,  approaches  have 
been  identified  for  problem  structuring  which  can  be  used  for  analysing  and  transforming  complex 
(or  even  ‘chaotic’)  situations  into  problem  abstractions  one  can  work  with.  These  approaches  have  to  a 
large  extent  been  grounded  in  theory  and  become  established  in  practice  [8].  Wider  acceptance  and 
employment  of  ‘soft’  OA  would  enable  the  defence  community  to  significantly  enhance  its  capability  to 
deal  with  complex,  high  priority  problem  issues. 

The  development  of  a  Code  of  Best  Practice  (CoBP)  for  ‘soft’  OA  is  intended  to  increase  ‘soft’  OA’s 
acceptance  within  the  military  and  defence-oriented  operational  analyst  communities.  It  will  provide 


Quote  from  report  [2]  of  the  Defence  Science  Board 
Advisory  Group  on  Defence  Intelligence: 

“OR  can  significantly  increase  the  decision-maker’s  insight 
and  improve  the  quality  of  debate  around  key  decision 
issues” 


Look  at  the  following  two  problematic  situations: 

1 .  Minimise  the  fuel  cost  of  a  regular  repair  resupply 
activity. 

->  ‘hard’  OA:  use  dynamic  programming  to  solve  it. 

2.  Advise  on  the  balance  of  maritime,  land  and  sea 
capabilities  for  2030. 

->  ‘soft’  OA:  but  how  would  you  do  it? 
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pragmatic  guidelines  to  enable  this  type  of  OA  to  be  practised  in  a  more  rigorous  and  scientific  manner, 
especially  by  those  who  are  new  to  this  type  of  approach. 

Practice  has  shown  that  ‘soft’  OA  adds  value  in  problem  structuring1,  option  design  and  action  planning. 
A  CoBP  should  ensure  that  the  value  of  ‘soft’  OA  is  maximised.  On  the  one  hand,  a  CoBP  that  addresses 
issues  of  validity  will  most  likely  increase  credibility  of  study  results  which  can  therefore  be  trusted  and 
accepted.  On  the  other  hand,  a  CoBP  that  also  addresses  what  a  client  can  and  should  expect  from  ‘soft’ 
OA,  how  to  assess  its  results  and  how  it  leads  to  recommendable  actions  will  also  most  likely  increase 
understanding  and  appraisal  of  the  relevance  of  such  approaches.  This  will,  in  the  end,  be  beneficial  to  the 
quality  of  defence  decision  making  and  the  value  and  versatility  of  OA  support,  both  in  direct  support  of 
operations  and  in  longer  term  support  of  (strategic)  defence  planning. 


1.2  READERSHIP  AND  ORGANISATION  OF  THE  GUIDE 

The  Guide  is  aimed  at  three  different  groups  of  readers : 

•  People  who  either  own  or  are  involved  in  the  problematic  situation  or  have  stakes  at  or  an  interest 
in  resolving  the  problematic  situation  and  people  who  are  otherwise  involved  in  the  process  as 
providers  of  viewpoints,  information,  and  other  aspects  of  relevance,  etc.  (i.e.  clients2  and  other 
stakeholders,  subject-matter  experts); 

•  People  who  conduct  the  study,  suggest  the  methodologies  and  methods  to  be  used,  facilitate  the 
study  process  or  otherwise  offer  analytical  support  (i.e.  operational  analysts,  facilitators)  or 
scrutinise  the  study  (i.e.  scrutineers);  and 

•  People  who  as  executive  or  commanding  officers  have  the  burden  of  the  final  responsibility  of 
having  a  problematic  situation  resolved,  who  have  to  agree  on  decision  recommendations  and 
actually  take  the  final  decision  (i.e.  executive  decision  makers). 

Under  the  common  header  of  “NATO  Guide  for  Judgement-Based  Operational  Analysis  in  Defence 
Decision  Making”  the  Guide  has  been  organised  in  three  separately  published  volumes  in  conformity  with 
the  three  types  of  readers  listed  above. 

1.2.1  An  Analyst-Oriented  Document 

This  document  carries  the  sub-title  “Code  of  Best  Practice  for  ‘Soft’  Operational  Analysis”  and  is  in  fact 
the  Code  proper  and  the  most  extensive  volume  of  the  three.  It  addresses  the  issues  that  the  NATO  Task 
Group  felt  would  be  important  for  analysts  to  know  when  new  to  the  field  of  ‘soft’  OA;  it  sets  the  ‘rules  of 
the  road’  for  analysts.  Later  sections  of  this  chapter  will  introduce  this  document  in  more  detail. 

1.2.2  A  Client-Oriented  Document 

This  is  a  document  which  explains  what  ‘soft’  OA  is  in  terms  of  judgement-based  analysis  and  what  one 
can  expect  from  it.  It  is  based  on  those  elements  of  the  first,  analyst-oriented  document  that  are  relevant  to 
clients.  The  document’s  chapters  address  the  following  questions: 


1  Although  a  large  body  of  literature  exists  about  what  is  known  as  ‘Problem  Structuring  Methods’  (which  encompass  some 
specific  methods/methodologies),  this  CoBP  has  adopted  the  view  that: 

a)  Problem  Structuring  Methods  can  do  more  than  merely  structure  problems;  and 

b)  There  are  quite  a  few  other  methods/methodologies  (e.g.  Multi-Criteria  Analysis)  that  also  enable  problem  structuring. 

This  CoBP  will  therefore  not  treat  Problem  Structuring  Methods  as  a  methodology  of  its  own  but  as  part  of  the  broader  family 
of  modelling  approaches  that  ‘soft’  OA  comprises. 

2 

Different  types  of  people,  their  roles  and  responsibilities  will  be  summarised  in  Section  1.6  and  discussed  further  in  Chapter  4. 
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•  What  is  judgement-based  OA? 

•  Which  problematic  situations  require  judgement-based  OA? 

•  How  does  judgement-based  OA  add  value? 

•  What  does  a  judgement-based  OA  study  look  like? 

•  What  is  expected  of  me,  the  client? 

•  What  does  the  analyst  bring  to  achieve  validity,  credibility  and  acceptance? 

•  How  can  a  CoBP  protect  the  client  from  threats  to  the  study? 

1.2.3  A  Document  for  the  High-Level,  ‘Executive’  Decision  Maker 

This  document  is  a  concise  brochure  carrying  the  sub-title  “Judgement-Based  Operational  Analysis  for 
Improved  Defence  Decisions”.  It  summarises  the  client-oriented  document  and  is  written  for  executive 
decision  makers,  including  high-ranked  officers,  and  addresses  the  following  issues: 

•  What  is  judgement-based  OA? 

•  How  can  defence  decision  makers  be  supported? 

•  When  should  judgement-based  OA  be  used? 

•  What  can  you  do  with  facilitated  workshops? 

•  What  are  the  value  and  the  benefits  of  judgement-based  OA? 

It  also  mentions  the  other  volumes  of  the  Guide  and  points  out  where  more  information  can  be  found. 


1.3  PURPOSE  OF  THE  CODE  OF  BEST  PRACTICE 

A  code  of  best  practice  captures  and  presents  the  peer  consensus  of  the  practitioners  in  a  specific  field  of 
the  behaviours,  processes,  information,  and  standards  which  constitute  the  current  recommendable  mode 
of  action  in  that  field.  It  aims  to  inform  and  guide  practitioners  in  its  field,  both  established  and  new,  on 
the  considered  best  ways  of  proceeding  so  as  to  complement  and  extend  the  existing  knowledge  base. 
It  also  defines  the  boundaries  of  accepted  best  practice,  indicating  where  knowledge  becomes  weak  or 
non-existent,  and  where  further  research  may  be  required  to  extend  the  code. 

A  code  of  best  practice  may  be  derived  from  a  consensus,  formal  or  informal,  or  could  be  the  product  of  a 
committee  or  a  single  mind.  It  gains  its  authority  from  peer  review  and  the  success  of  its  application. 
The  CoBP  for  ‘soft’  OA  aims  to  inform,  guide,  validate,  make  credible  and/or  indoctrinate  current  practices 
for  readers  working  to  solve  problems  that  require  using  ‘soft’  OA  methods  and  techniques. 

The  CoBP  for  ‘soft’  OA  aims  to  provide  practical  advice,  and  to  set  a  benchmark  for  best  practice  to  both 
clients  and  analysts  in  order  to  get  the  best  out  of  the  resources  available.  There  should  be  some 
expectation  that  the  analyst  community  uses  this  Code  as  guidance  for  best  practice  use  of  ‘soft’  OA 
methods  (this  volume).  A  client  should  use  the  Code  to  enable  a  better  understanding  of  how  ‘soft’  OA 
methods  can  help  the  client  to  make  effective  decisions  (client-oriented  volume,  printed  under  separate 
cover).  The  Code  aims  to  help  an  analyst  and  client  cope  with  the  issues  of  subjectivity  that  come  with 
using  ‘soft’  OA  methods,  and  to  cope  with  any  analyst/client  bias  in  the  analysis  process.  It  aims 
to  provide  guide  to  practitioners  concerning  the  quality  of  subjective  infonnation  and  analysis, 
the  transparency  of  the  methods  used  and  the  credibility  of  the  outputs. 

The  purpose  of  the  CoBP  is: 

•  To  create  an  understanding  of  what  ‘soft’  OA  is; 
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•  To  clarify  what  ‘soft’  OA  can  do  to  help  address  problematic  situations,  and  what  people 
can  expect  from  it  in  that  respect;  and 

•  To  provide  guidance  on  how  a  ‘soft’  OA  study  should  be  carried  out  in  order  to  maximise 
the  validity,  credibility  and  acceptance  of  the  study  and  its  outcomes. 

The  CoBP  is  designed  to  be  educational  in  the  sense  that  it  aims  to  clarify  what  ‘soft’  OA  is,  and  when  and 
how  it  can  or  should  be  used.  The  analyst-oriented  volume,  the  actual  Code,  sets  the  rules  of  the  road  for 
analysts  who  are  relatively  new  to  this  field  of  OA.  It  is  however  expected  that  more  experienced  ‘soft’ 
OA  analysts  will  also  find  it  of  value  for  educational  of  reference  purposes  whilst  having  developed  their 
own  views  on  how  this  type  of  analysis  should  be  carried  out  and  perhaps  their  own  methods  or 
methodologies.  The  guidelines  are  offered  as  a  means  of  assistance  to  analysts  and  are  not  meant  to  be  too 
prescriptive  in  nature. 

1.4  SCOPE  OF  THE  CODE  OF  BEST  PRACTICE 

The  purpose  of  decision  support  by  ‘soft’  OA  remains  the  primary  scope  of  the  CoBP.  Within  that  context, 
the  CoBP  will  look  at  ‘soft’  OA  across  the  whole  spectrum  of  applications.  The  CoBP  will  therefore  deal 
with  problem  types  that  require  a  decision  at  some  point,  be  it  by  an  individual  or  by  a  group  of  people, 
be  it  a  final  decision  or  perhaps  rather  a  recommendation  for  a  decision,  be  it  a  short-term  decision  on,  for 
example,  operational  contingencies  or  a  long-term  impact  decision  regarding  some  strategic  issue.  Typically, 
a  great  many  workshop  sessions  are  held  and  facilitated  where  a  group  of  people  meet  to  identify  relevant 
knowledge  or  discuss  some  other  issue  of  interest  without  identifying  options  that  have  to  be  decided  upon. 
However,  the  CoBP  will  not  address  single  session  studies  for,  for  example,  mere  knowledge  elicitation. 
Facilitation  methods  exist  which  could  be  regarded  as  ‘soft’  and  may  be  helpful  for  this  type  of  limited-scope 
workshops,  but  these  are  not  the  types  of  studies  or  workshops  that  the  CoBP  addresses.  The  CoBP  is  not 
meant  to  be  a  guide  to  facilitation  techniques  or  the  organisation  of  workshops. 

The  CoBP  does  not  describe,  compare  or  assess  individual  ‘soft’  methods  and  techniques.  There  are 
textbooks  and  articles  that  do  that.  The  CoBP  is  rather  focused  on  issues  of  their  usage  and  contribution  to 
rigorous  and  auditable  quality  of  both  process  and  content.  It  does  not  recommend  any  of  the  existing 
formal  methodologies  or  methods  other  than  in  a  general  sense.  However,  ideas  of  how  to  approach 
specific  aspects  have  been  adopted  from  existing  (refereed)  literature.  Chapters  6  and  7  will  mention  some 
methods,  by  way  of  examples,  to  address  specific  issues. 

Further,  as  stated  previously,  studies  that  require  ‘hard’  OA  usually  have  aspects  that  require  human 
judgement  and  thus  may  benefit  from  ‘soft’  OA  methods.  The  CoBP  does  not  address  ‘soft’  aspects  of 
‘hard’  studies  (e.g.  interpreting  ‘hard’,  numerical  model  results)  as  a  particular  phenomenon  in  itself. 
If  those  ‘soft’  aspects  are  substantial  and  are  perhaps  even  supported  by  a  method,  then  one  is  dealing  with 
a  mixed  (‘multi-methodology’)  approach  (where  ‘soft’  and  ‘hard’  methods  are  combined)  which  will  be 
discussed  in  Chapter  6. 

The  use  of  OA  in  an  operational  setting  is  explored  by  the  NATO  Technical  Team  SAS-089  “Operational 
Analysis  Support  to  NATO  Operations”.  Its  aim  is  to  elaborate  a  recommendation  of  an  organisational 
structure,  common  procedures,  technical  training,  and  tools  that  improve  effectiveness,  interoperability, 
and  sustainability  of  OA  in  support  to  operations  in  both  NATO  and  national  static  and  deployed  HQs. 
The  focus  of  the  work  of  the  team  is  on  the  provision  of  direct  -  especially  deployed  -  OA  support  to 
NATO  Operations.  The  purpose  of  such  support  is  to  improve  the  quality  and  pace  in  the  decision-making 
process  during  all  phases  of  operations.  This  subject  will  therefore  not  be  addressed  in  this  CoBP. 

For  pragmatic  reasons  the  CoBP  has  adopted  the  term  ‘soft’  even  though  it  may  be  argued  that  this 
incorrectly  suggests  that  a  clear  distinction  between  ‘soft’  and  ‘hard’  exists.  It  has  already  been  argued, 
and  to  be  elaborated  in  Chapter  2,  that  even  a  ‘hard’  study  has  ‘soft’  elements.  Similarly,  such  terms  as 
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‘qualitative’  and  ‘quantitative’,  ‘objective’  and  ‘subjective’  suggest  distinctions  that  are  not  always 
warranted  in  practice.  Initially,  the  authors  of  the  CoBP  attempted  to  introduce  the  term  ‘judgement-based’ 
OA,  since  what  the  CoBP  is  about  to  describe  is  a  type  of  analysis  that  is  predominantly  based  on  human 
judgement.  Judgement-based  OA  applies  the  methods  of  ‘Soft  Operational  Research’  developed  in 
academia.  It  has  appeared,  however,  that  the  analytic  community  has  become  used  to  the  term  ‘soft’  OA  to 
such  a  degree  that  the  formal  use  of  the  term  ‘judgement-based’  OA  would  have  created  unnecessary 
confusion.  The  CoBP  uses  ‘judgement-based’  where  a  special  emphasis  on  the  use  of  judgement  is 
needed. 


1.5  OVERVIEW  OF  THE  CODE  OF  BEST  PRACTICE 

The  chapters  of  the  CoBP  are  organised  according  to  four  main  themes  put  forward  in  the  form  of  the 
following  questions  (ref.  Figure  1-1): 

•  What  is  the  CoBP  about? 

•  Why  was  it  written? 

•  Who  is  involved  in  a  ‘soft’  OA  study  and  what  is  expected  of  them? 

•  How  should  a  ‘soft’  OA  study  be  conducted? 


Chapter  1  -  INTRODUCTION 

Chapter  2  -  PROBLEMATIC  SITUATIONS  AND  'SOFT'  OA 
Chapter  3  -  ACHI EVI NG  VALI DITY,  CREDI  Bl  LITY  AND  ACCEPTANCE 


Chapter  4  -  ROLES  AND  RESPONSIBILITIES 


•  Chapter  5  -  THE  ‘SOFT’  OA  STUDY  PROCESS 

•  Chapters  -  DESIGNING  THE 'SOFT' OA  STUDY  METHODOLOGY 

•  Chapter  7  -  DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 

•  Chapter8  -  INTERPRETING,  USING  AND  COMMUNICATING  RESULTS  AND  OUTCOMES 


Chapter  9  -  THE  ESSENCE  OF  THE  CODE  OF  BEST  PRACTI CE 

Annex  A  -LIST  OF  KEY  STATEMENTS  AND  RECOMMENDATIONS  OF  IMPORTANCE 
TO  THE  ANALYST 

Annex  B  -  SELECTIVE  LIST  OF  'SOFT'  METHODS 


Figure  1-1:  Line  of  Reasoning  Concerning  the  Flow  of  the  Chapters. 


Below  follows  an  overview  of  the  content  of  each  chapter  and  the  annexes  of  the  CoBP: 
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•  Chapter  1  presents  an  introduction  to  the  subject,  including  scope,  purpose,  and  meaning  of  the 
CoBP  as  well  as  some  information  on  the  working  procedure.  It  also  provides  an  overview  of 
some  key  concepts  that  are  used  throughout  the  CoBP;  many  of  these  will  be  elaborated  in  specific 
subsequent  chapters. 

•  Chapter  2  provides  a  general  introduction  to  issues  that  are  pertinent  to  the  nature  of  ‘soft’  and 
‘hard’  OA,  including  the  phenomenon  of  uncertainty.  It  attempts  to  characterise  ‘soft’  and  ‘hard’ 
OA  rather  than  provide  a  formal  definition  for  each  term.  This  chapter  also  addresses  why  one 
would  do  ‘soft’  OA  in  the  first  place. 

•  Chapter  3  discusses  how  a  ‘soft’  study  contributes  to  addressing  a  problematic  situation  as  intended 
by  the  CoBP.  It  shows  the  line  of  reasoning  of  the  CoBP  and  the  philosophy  of  the  use  of  methods, 
with  a  particular  emphasis  on  aspects  related  to  the  purpose  of  writing  the  CoBP  (e.g.  validity, 
credibility  and  acceptance).  This  will  be  addressed  again  in  later  chapters,  but  more  specifically  in 
Chapter  8  where  issues  of  interpretation  and  the  use  of  study  outcomes  are  discussed. 

•  Chapter  4  presents  a  discussion  of  the  roles  and  responsibilities  of  the  various  actors  involved  in  a 
‘soft’  OA  study,  with  a  particular  emphasis  on  the  analyst  and  the  facilitator.  The  use  of  subject- 
matter  experts  will  be  discussed  in  Chapter  7.  Study  success  and  acceptance  largely  depend  on  the 
quality  of  the  process  wherein  the  various  actors  inteiplay,  and  depends  to  a  large  degree  on 
procedural  rationality  (i.e.  the  quality  of  how  things  are  done  and  organised)  in  conjunction  with 
substantive  rationality  (i.e.  the  quality  of  what  is  done). 

•  Chapter  5  presents  an  overview  of  the  stages  of  the  study  process  when  conducting  a  ‘soft’  OA. 
It  is  an  iterative  cycle  where  products  or  findings  of  previous  stages  are  reviewed  based  on  new 
insights  gained  during  later  stages.  The  way  in  which  the  process  is  set  up  and  actually  conducted 
does  in  itself  contribute  to  the  validity,  credibility  and  acceptance  of  a  study.  The  chapter  also 
addresses  issues  of  parallel  and  sequential  modelling  activities,  and  divergent  and  convergent 
process  stages. 

•  Chapter  6  addresses  the  modelling  process  in  general  and  ‘multi-methodology’  in  particular, 
which  is  the  approach  where  various,  possibly  very  different,  methods  are  combined  in  one  study. 
It  provides  guidelines  as  to  the  suitability  of  a  combined  approach  and  when  to  apply  it.  Many  of 
the  guidelines  are  also  relevant  to  a  single-methodology  approach.  The  chapter  also  discusses  the 
initial  stages  of  a  study  where  the  nature  of  the  problematic  situation  has  to  be  assessed  and  upon 
which  the  design  of  the  study  methodology  must  be  based. 

•  Chapter  7  presents  an  overview  of  the  ways  in  which  data  can  be  collected,  including  elicitation 
of  information  from  individual  experts,  sensitivity  analysis,  coping  with  data  uncertainty, 
and  dealing  with  subject-matter  experts  who  are  asked  to  provide  information  in  a  facilitated 
workshop  setting.  This  chapter  draws  heavily  from  Chapter  4,  as  people  (even  experts)  are  often 
subject  to  biases  when  answering  questions  It  stands  in  its  own  right  as  far  as  facilitation  of 
groups  of  people  is  concerned,  how  data  is  gathered  and  interpreted  using  subject-matter  experts, 
and  how  people  work  together  to  produce  outcomes. 

•  Chapter  8  addresses  how  to  communicate  and  report  on  the  outcomes  of  a  ‘soft’  OA  study. 
This  chapter  draws  on  Chapter  7  about  framing  and  biases,  but  also  on  Chapter  3  as  far  as  validity, 
credibility  and  acceptance  are  concerned.  Triangulation,  sensitivity  analysis  and  coping  with  bias 
all  contribute  to  the  validity  of  the  results  and  are  discussed  in  this  chapter  from  a  somewhat 
different  perspective  than  previous  chapters. 

•  Chapter  9  concludes  with  a  summary  of  the  essence  of  the  CoBP  and  some  final  remarks. 

•  Annex  A  contains  a  list  of  key  statements  and  recommendations  that  are  taken  from  the  initial 
summarising  text  boxes  of  each  chapter  and  are  especially  important  to  the  analyst  (the  ‘rules  of 
the  road’). 
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•  Annex  B  contains  a  selective  list  of  ‘soft’  OA  methods  and  methodologies,  quoted  from  [9]. 


1.6  KEY  CONCEPTS 

It  is  useful  to  discuss  some  key  concepts  that  permeate  throughout  the  CoBP.  The  definitions  are  based  on 
[10],  and  more  specifically  [11],  but  have  been  adapted  for  the  purpose  of  the  CoBP: 

•  ‘ Methodology '  -  Although  the  general  philosophical  meaning  is  that  of  ‘the  study  of  methods’, 
the  CoBP  will  regard  ‘methodology’  as  ‘ a  particular  combination  of  methods  that  possess  a 
common  set  of  characteristics  and  assumptions  and  are  used  in  a  study  to  achieve  a  purpose ' . 
That  purpose  may  be  the  overall  aim  of  the  OA  study  or  a  part  of  it. 

•  ‘ Method ’  -  This  term  will  be  regarded  as  ‘a  structured  set  of  guidelines  or  rules  to  achieve  some 
clear  well-defined  purpose ’  which  may  or  may  not  involve  mathematical  calculations.  Here, 
‘purpose’  is  meant  to  be  a  part  of  the  aim  of  the  study  and  its  achievement  will  contribute  to  the 
overall  study  aim.  The  rules  may  be  algorithmic  in  the  mathematical  sense,  but  could  also  be 
behaviour-  or  procedure-oriented,  or  a  combination  of  both. 

•  ‘‘Technique’  or  ‘Tool’  -  These  terms  denote  each  of  the  specific  algorithms  or  interviewing  modes 
(etc.)  which  are  part  of  a  method.  A  difference  could  be  made  between  the  two  where  ‘tool’  would 
be  regarded  as  the  agent  of  a  technique,  e.g.  a  software  program. 

•  ‘Model'  -  A  model  will  be  regarded  as  ‘a  representation  of  (a  part  of)  reality  (i.e.  a  problematic 
situation)  as  seen  by  a  number  of  people  who  wish  to  use  the  model  to  understand,  manage, 
or  improve  that  reality' .  The  model  may  include  the  beliefs,  insights  and  expectations  of  people 
who  wish  to  use  the  model  as  a  means  of  communication  and  stimulus  of  debate  and  reflection 
(i.e.  support  their  thinking)  to  create  or  improve  a  mutual  understanding  of  each  other’s 
viewpoints  and  positions.  Both  setting  up  a  model  and  using  it  (or,  in  ‘hard’  OA  terms,  even 
solving  it)  usually  requires  a  method. 

The  definitions  above  suggest  a  clear  difference  between  ‘method’  and  ‘methodology’.  In  practice,  that 
difference  is  blurred  and  the  two  are  used  inter-changeably  in  the  literature;  there  is  no  clear  agreement  in 
the  literature  about  these  terms.  In  the  CoBP  ‘method’  will  be  used  to  denote  specific  methods  and 
‘methodology’  will  be  used  to  denote  a  combination  of  methods  or  a  general  study  approach  where  one  or 
more  methods  are  used. 

In  the  light  of  the  definitions  above,  ‘soft’  OA  may  be  regarded  as  a  ‘ meta-methodology' :  a  higher  level  of 
methodology,  encompassing  all  ‘soft’  methods  and  methodologies.  Note  the  distinction  from  the  term 
‘ multi-methodology '  which  is  a  specific  combination  of  different  methodologies  used  in  a  study.  A  multi¬ 
methodology  approach,  based  on  a  mix  of  appropriate  (‘hard’  and/or  ‘soft’)  methods,  has  appeared  to  be 
very  common  and  successful  in  practice  and  will  be  addressed  in  Chapter  6. 

There  are  a  number  of  additional  key  concepts  that  need  to  be  discussed  at  the  outset: 

•  ‘Intervention'3  -  This  term  denotes  the  process  of  challenging  and  changing  people’s  thinking  or 
an  organisation’s  activities  in  order  to  achieve  understanding  and  improvement  and  often,  along 
the  way,  resolving  problematic  situations.  The  CoBP  will  avoid  using  intervention  but  rather  use 
‘study’  instead. 

•  ‘Study'  -  This  term  denotes  a  set  of  activities  that  is  part  of  an  intervention  and  usually  refers  to 
more  specific  actions  of  investigation  and  research,  problem  analysis  and  resolution. 


3 

‘Intervention’  will  not  be  regarded  as  a  military  operation  like  in  ‘the  intervention  in  Afghanistan’  unless  specifically  indicated 
as  such. 
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•  "Analysis’’  -  This  term  denotes  a  set  of  activities  aimed  at  decomposing  (physical,  organisational, 
social)  systems,  problems,  problematic  situations,  operations,  etc.,  into  their  constituent  parts,  and 
at  investigating  their  relationships  and  their  meaning. 

•  ‘ Operational  Analysis'  (term  mostly  used  in  the  defence  community,  but  equivalent  to  the  term 
‘operational  research’  used  in  academia)  is  the  interdisciplinary  science  that  focuses  on  how 
appropriate  actions  can  be  designed  to  change  (i.e.  towards  improvement)  or  even  (re)solve 
problematic  situations.  The  CoBP  addresses  ‘soft’  OA,  an  approach  that  uses  methodologies  and 
methods  predominantly  based  on  the  rational  (i.e.  not  intuitive)  use  of  human  judgement. 

•  ‘ Result ’  -  This  term  denotes  what  has  been  produced  by  using  a  model  and  applying  a  method,  or 
merely  following  a  process.  It  is  rather  more  specific  than  ‘outcome’.  ‘Results’  constitute  the 
ingredients  for  an  ‘outcome’  (to  be  analytically  interpreted),  although  in  practice  the  distinction  is 
not  always  crisp. 

•  ‘ Outcome ’  -  This  term  denotes  what  has  been  achieved  with  the  study  (‘effect’  or  ‘impact’),  based 
on  specific  results  and  their  analytic  interpretation.  It  refers  to  what  one  can  do  with  the  results 
and  what  the  meaning  of  the  results  is. 

•  ‘ Decision ’  -  This  term  refers  to  the  result  of  making  up  one’s  mind  regarding  a  choice  between 
alternatives.  The  kinds  of  decisions  that  will  be  taken  on  the  basis  of  a  ‘soft’  OA  study  pertains  to 
a  choice  between  alternatives:  ways  forward  (‘actions’),  (organisational)  strategies,  policy 
options,  and  the  like. 

•  ‘ Alternative ’  or  ‘ Option ’  -  This  term  denotes  one  of  two  or  more  possibilities  to  be  chosen  for 
addressing,  improving  or  (re)solving  a  problematic  situation. 

•  ‘ Stakeholder  4  -  This  term  denotes  the  person  who  affects  or  is  affected  by  the  (resolution  of  the) 
problematic  situation.  It  encompasses  rather  different  types  of  people  who  are,  or  have  to  be, 
involved  in  a  study.  Even  ‘analysts’  and  ‘facilitators’  (see  below)  may  to  some  extent  be  regarded 
as  ‘stakeholders’  as  they,  too,  affect  the  study  or  are  affected  by  it,  e.g.  their  interest  in  its 
perceived  success.  The  CoBP  will  address  the  way  in  which  stakeholders  should  be  involved  in  a 
study,  and  what  roles  they  play. 

•  ‘ Client ’  -  This  term  denotes  four  specific  types  of  stakeholders:  the  sponsor  who  actually  owns 
the  study,  the  customer  who  pays  the  bill  for  the  study,  the  decision  maker  who  may  make 
decisions  regarding  the  problematic  situation  and  the  end  user  who  is  ultimately  affected  (either 
positively  or  negatively)  by  the  study  outcomes  and  related  decisions.  All  four  together  own  in 
some  way  the  problematic  situation.  The  real  client  could,  however,  remain  invisible,  and  could 
indeed  be  unknown.  Note  that  situations  can  exist  where  there  is  not  one  single  client  but  rather  a 
group  of  clients  (‘client  system’)  who  may  have  (very)  different  interests  in  or  views  of  the 
problematic  situation,  even  if  formally  they  adopt  the  same  role.  The  ‘primary  client’  is  the  first 
point  of  contact  of  the  analyst;  primary  client  and  analyst  are  partners  throughout  the  study. 

•  ‘ Subject-Matter  Expert ’  -  This  term  denotes  an  individual  who  has  considerable  relevant 
knowledge  without  necessarily  owning  or  otherwise  being  part  of  the  problematic  situation.  He4 5  is 
the  expert  in  a  particular  area  or  topic  and  provides  infonnation;  in  doing  so,  he  affects  the  study. 
The  CoBP  will  specifically  address  issues  concerning  the  elicitation  of  information  from  experts. 

•  ‘ Analyst ’  -  This  term  denotes  a  person  who  conducts  a  study,  and  in  this  capacity  designs  the  stages 
of  the  process,  suggests  the  methodology(ies)  and  method(s)  to  be  used,  the  workshops  to  be  held, 
ways  to  analyse  and  model  the  problematic  situation  and  to  report  and  interpret  the  study’s 
outcomes.  The  analyst  should  work  in  cooperation  with  the  client(s)  and  with  a  fundamentally 
helpful  attitude,  but  at  the  same  time  he  should  be  devoted  to  rigorous  and  ethical  principles. 


4  The  types  of  people  that  follow,  including  their  roles  and  responsibilities,  will  be  discussed  further  in  Chapter  4. 

5  Throughout  the  CoBP  the  male  fonn  is  used  for  simplicity’s  sake.  This  does,  of  course,  in  no  way  rule  out  any  female  actors. 
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•  ‘ Facilitator ’  -  This  term  denotes  a  person  who  helps  (i.e.  supports,  enables  and  encourages) 

individuals  or  a  group  or  groups  of  stakeholders,  clients,  subject-matter  experts,  and  analysts  work 
together  through  the  stages  of  a  study  by  managing  procedure  (the  way  the  problem  is  tackled) 
and  process  (the  way  participants  interact),  whilst  adopting  an  impartial  attitude.  Facilitation  is, 
similar  to  analysis,  fundamental  to  the  CoBP  as  the  quality  of  a  ‘soft’  OA  study  depends  on  it. 
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>  Three  types  of  a  problematic  situation  can  be  distinguished:  puzzle,  problem  (proper),  mess.  Each 
type  is  approached  somewhat  differently.  ‘Soft’  OA  tends  to  be  more  suitable  for  problems  (proper) 
and  messes. 

>  ‘Hard’  and  ‘soft’  OA  both  support  decisions  regarding  problematic  situations.  There  are  however 
numerous  differences  between  them  from  a  large  number  of  perspectives,  although  in  practice  each 
perspective  represents  a  spectrum. 

>  Uncertainty  is  one  of  the  key  phenomena  that  need  to  be  addressed  in  a  problematic  situation  as  it 
characterises  a  problematic  situation  to  a  large  extent. 

>  ‘Soft’  OA  aims  to  structure  problematic  situations  (with  a  focus  on  problems  proper  and  messes), 
identify  stakeholders  and  their  interests  and  perceptions,  include  them  in  the  study,  use  modelling 
primarily  as  a  means  to  clarify  issues  and  enhance  communication  but  also  to  identify  possible  options 
for  a  most  preferred  way  ahead  in  resolving  the  problematic  situation. 

>  Messy  problems  cannot  be  fully  resolved  but  should  rather  be  managed. 

>  ‘Soft’  and  ‘hard’  OA  are  in  many  respects  complementary  to  one  another;  they  are  not  competing  OA 
approaches. 
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2.1  INTRODUCTION 

This  chapter  introduces  three  types  of  a  problematic  situation  that  may  be  encountered.  It  then  discusses 
the  characteristics  of  ‘hard’  and  ‘soft’  OA  and  points  out  similarities  and  differences.  It  focuses  on  the 
nature  of  data  and  models  (and  methods)  and  relates  this  to  ‘soft’  and  ‘hard’  OA.  The  chapter  also  discusses 
the  phenomenon  of  uncertainty  that  to  a  large  extent  characterises  the  type  of  problematic  situations 
addressed  by  OA  (and  specifically  ‘soft’  OA  in  a  defence  context).  This  chapter  ends  by  summarising  why 
one  would  use  ‘soft’  OA  in  a  study. 


2.2  PROBLEMATIC  SITUATIONS 

The  term  ‘problem’  has  many  meanings:  it  is  used  in  this  CoBP  for  any  situation  in  which  there  is  a 
perceived  gap  between  its  current  state  and  its  desired  or  required  state,  and  which  needs  or  seems  to  need  a 
(re)solution  involving  some  action,  although  it  may  often  be  unclear  what  (re)solution  and  what  action  are 
most  appropriate.  The  term  also  has  a  specific  meaning  as  one  type  of  ‘problematic  situation’  (see  below). 

For  the  purpose  of  the  CoBP  it  is  useful  to  distinguish  between  a  few  distinct  problematic  situations. 
The  reason  for  this  is  that  they  usually  require  different  approaches  to  address  them.  The  literature 
suggests  the  following  three  problematic  situation  types  summarised  in  Table  2-1  (ref.  e.g.  [1]): 

•  ‘ Puzzles'  -  These  are  problematic  situations  where  the  issue  is  clear,  what  needs  to  be  achieved  is 
clear  and  the  way  in  which  this  should  be  done  (i.e.  the  model  to  be  set  up  and  the  method  to  be 
applied)  is  also  clear.  Even  in  the  case  where  these  elements  are  not  immediately  clear, 
an  agreement  on  their  definition  can  be  easily  reached.  Achieving  a  solution  to  a  ‘puzzle’  may  still 
be  complicated  but  knowing  how  to  build  the  model  and  which  method  to  apply  is  a  matter  of 
education  and  expertise.  The  quality  of  the  solution  (e.g.  its  optimality)  is  usually  testable. 

•  ‘ Problems’’  -  In  this  CoBP,  problems  have  a  more  specific  meaning  and  will  therefore  referred  to 
as  ‘problem  (proper)’.  They  denote  situations  where  the  issue  may  be  more  or  less  clear  and 
structured  and  what  needs  to  be  achieved  is  (partially)  clear  as  well.  It  is  however  not  clear  at  the 
outset  in  what  way  a  solution  to  a  problem  should  be  designed.  Whereas  the  issue  and  the  desired 
achievement  may  be  readily  agreed  upon,  achieving  a  solution  is  not  a  routine  activity  but  will 
have  to  be  negotiated  depending  on  the  views  of  the  people  (including  analysts)  involved  and 
(often  many)  perspectives  that  are  relevant.  Therefore,  it  is  not  likely  that  an  overall  optimum 
solution  will  be  produced.  Rather  a  ‘most  preferred’  (or  ‘satisficing’1)  solution  or  set  of  solution 
options  will  be  the  result  of  the  analysis,  which  will  therefore  require  creativity  in  addition  to 
education  and  expertise. 

•  ‘ Messes’  (or:  ‘  Wicked  Problems' )2  -  These  are  problematic  (or  even  chaotic)  situations  that  are 
ill-structured  or  do  not  seem  to  have  any  structure  at  all.  The  issues,  and  how  they  are  interrelated, 
are  not  clear  and  considerable  disagreement  exists  about  them.  Similarly,  there  is  disagreement 
regarding  what  is  to  be  achieved,  who  is  to  be  involved  and  the  steps  that  are  to  be  taken  towards 
some  improvement  or  acceptable  change  of  the  situation.  What  is  assumed,  however,  is  that  to 
some  degree  and  with  at  least  some  people  a  sense  of  concern  exists  about  the  situation;  something 
needs  to  be  done,  some  decision  must  be  taken  in  the  end.  What  is  also  assumed  is  that  it  is 
possible  to  argue  over  the  definition  of  the  characteristics  of  the  problematic  situation.  Messes  are 
not  solved,  but  managed  [2]:  coping  with  ‘messes’  requires  considerable  ingenuity  and  both 
analytical  and  social  skills  in  addition  to  expertise  with  similar  situations.  No  two  ‘messes’  are  the 
same. 


1  A  decision  or  strategy  or  (re)solution  is  satisficing  if  it  is  good  enough  (‘adequate’),  but  not  necessarily  optimal. 

2 

Rittel  and  Webber  formally  introduced  the  term  ‘wicked  problem’  in  [4],  although  they  had  first  used  it  in  the  1960s.  Ackoff 
[5]  introduced  the  tenn  ‘mess’  or  ‘messy  problem’.  Although  formally  not  identical,  for  practical  purposes  within  this  CoBP 
they  denote  a  similar  problematic  situation  type.  This  CoBP  uses  the  term  ‘mess’  or  ‘messy  problem’. 
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Table  2-1:  Main  Characteristics  of  Puzzles,  Problems  and  Messes. 


Puzzle 

Problem 

Mess 

Description 

Well-defined  issue  with  a 
specific  solution  that  can 
be  worked  out. 

Well-defined  issue,  but 
with  no  single  solution  or 
approach. 

Complex  issue  which  is  not 
well  defined. 

Formulation  of  the 

Issues 

Agreed  objectives 
achieved  through  logical 
analysis. 

Agreeable  after  negotiation 
between  clients  and 
analysts,  and  input  by 
other  interested  parties. 

The  lack  of  clarity  and 
agreement  of  the  issues 
themselves  will  make 
formulation  disputed. 

Typical  Advice 

Optimal  solution. 

Preferred  option  based  on 
ranking. 

Possible  courses  of  action. 

Validity  of  the  Advice 

Accepted  after  scrutiny  of 
mathematics  and 
calculations. 

Even  though  the  results 
are  reached  by  agreed 
methods,  the  conclusions 
can  be  disputed  thus 
leading  to  informed  debate 
about  the  actions  to  take. 

The  advice  will  lead  to  an 
informed  debate  about  the 
pros  and  cons  of  several 
possibilities  for  the  way 
ahead. 

The  problems  or  problematic  situations  that  ‘soft’  OA  seeks  to  address  are  usually  the  messy  problems, 
and  to  some  extent  problems  proper,  that  present  no  obvious  solutions.  Problem  framing  is  thus  the  activity 
of  trying  to  understand  a  problematic  situation  so  that  meaningful  action  can  be  taken  (ref.  Chapter  6). 

When  approaching  a  messy  problem,  it  is  usually  clear  that  there  is  (at  least  initially)  no  agreed  problem 
definition.  Different  stakeholders  will  have  legitimately  different  and  perhaps  even  contradictory  views  of 
its  nature.  In  this  context,  problems  have  to  be  seen  as  socially  constructed  [3].  This  means  that  there 
cannot  be  a  correct  problem  definition,  merely  many  possible  definitions  that  may  be  useful  for  facilitating 
action.  Moreover,  the  stated  problem  may  not  be  the  real  or  full  problem  and  the  problem  statement  should 
therefore  be  explored. 

A  well-defined  problem  is  helpful  for  conducting  a  study,  but  in  many  cases  it  may  not  be  possible  to  agree 
on  a  problem  definition.  In  order  to  get  a  useful  answer,  it  helps  if  you  know  what  the  question  is. 
At  the  same  time,  the  socially  constructed  nature  of  problems  means  that  getting  a  useful  problem  definition 
has  to  involve  some  kind  of  negotiation.  Often  in  ‘soft’  OA,  in  order  to  frame  a  problem,  a  facilitated  process 
is  used.  Many  of  the  more  well-established  methods  in  ‘soft’  OA  are  focused  on  problem  definition. 

It  has  also  been  argued  (Chapter  1)  that  ‘soft’ 

OA  will  be  of  value  in  decision-making 
situations  where  the  inputs  in  terms  of  data 
and  information  are  not  readily  or  reliably 
available3,  the  decision  consequences  cannot 
be  easily  foreseen,  or  where  decision  options 
may  even  be  undefined  and  clear  inputs  non¬ 
existent.  Also  for  this  reason,  ‘soft’  OA  and 
therefore  reliance  on  human  judgement  will 
most  likely  be  of  value  when  facing  ‘messes’ 
and  perhaps  ‘problems’  as  well.  ‘Hard’  OA 
will  most  likely  be  of  value  with  ‘puzzles’ 
and  to  a  lesser  degree  with  ‘problems’. 


In  NATO  a  capability  manager  was  asked  to  consider  the 
future  capabilities  that  might  be  needed  for  expeditionary 
operations.  He  took  the  NATO  definition  of  Expeditionary 
Operations  as  his  starting  point,  however  he  found  it  vague 
and  all-encompassing  and  therefore  not  useful  to  direct  or 
inform  capability  requirements  analysis.  Help  was  needed 
to  bound  and  more  clearly  define  expeditionary  operations 
before  a  study  of  requirements  could  begin. 
Morphological  analysis  was  used  to  identify  common 
understanding  about  the  key  characteristics  for 
expeditionary  operations  across  various  communities 
within  NATO  It  was  ideal  as  a  tool  to  deal  with  the  multi¬ 
dimensional  and  non-quantifiable  nature  of  the  problem. 

The  study  resulted  in  a  more  detailed  description  of 
expeditionary  operations  that  could  be  used  as  the  basis 
for  further  requirements  work. 


3 

This  non-availability  of  data  or  its  lack  of  reliability  may  also  occur  in  a  ‘hard’  OA  study.  In  the  latter,  one  would  most  likely 
resort  to  a  parametric  study. 
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Figure  2-1  illustrates4  that  puzzles  represent  clear-cut  problematic  situations  where  sufficient  factual  and 
complete  information  exists  and  little  judgement  is  required.  By  contrast,  messes  represent  complex 
problematic  situations  with  incomplete  and  little  factual  information  requiring  substantial  judgement. 
The  potential  usefulness  of  applying  ‘soft’  OA  increases  when  moving  from  the  upper  left  comer  of  the 
diagram  to  its  right  lower  comer. 


clear-cut 
nature  of 
situation 


more  clear-cut 
situation  requiring  less 
judgement 


reliance  on  human 
judgement 


Figure  2-1:  A  Spectrum  of  Problematic  Situations  (Problem  Types)  and 
Their  Clear-Cut  Nature  versus  Reliance  on  Human  Judgement. 


2.3  ATTITUDES  TOWARDS  UNCERTAINTY 

Defence  problems  almost  always  involve  many  kinds  of  uncertainty  from  a  range  of  sources.  Some  of  the 
uncertainty  emanates  from  the  characteristics  of  defence  issues,  and  some  of  it  is  generally  associated  with 
the  messy  nature  of  problematic  situations.  For  long-term  defence  issues,  the  main  uncertainty  is  the 
development  of  the  future  operational  environment,  which  determines  the  conditions  within  which  a  future 
capability  is  to  be  used.  As  new  military  capabilities  require  long  lead-times,  addressing  this  uncertainty  is 
challenging.  For  near-term  operational  issues  the  key  uncertainty  is  related  to  the  intentions  and  capabilities 
of  various  actors  in  a  complex  conflict  situation,  and  the  character  of  the  conflict  in  general. 

Uncertainty  in  itself  may  call  for  ‘soft’  OA.  In  addition,  messy  problems  show  other  forms  of  uncertainty. 
There  is  uncertainty  regarding  the  objectives  of  action,  closely  connected  to  the  uncertainty  of  intentions 
mentioned  above.  There  is  also  uncertainty  regarding  the  consequences  of  an  action,  emanating  from  a  limited 


4  See  [6]  for  an  alternative  way  of  illustrating  this. 
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understanding  of  cause-effect  relationships  between  factors  of  influence  and  unclear  or  debatable  boundaries 
of  problematic  situations. 

Fundamentally,  there  are  two  meanings  of  uncertainty  which  are  related  to  its  root  causes:  the  absence  of 
knowledge  and  the  inherent  variability  of  phenomena.  In  order  to  cope  with  uncertainty,  three  basic  attitudes 
towards  it  may  be  distinguished,  depicted  in  Figure  2-2.  The  comers  of  the  triangle  in  Figure  2-2  represent 
basic  ‘primitive’  attitudes,  or  types  of  bias,  towards  uncertainty: 

•  Prediction  is  the  attitude  that  one  should  attempt  to  minimise  uncertainty  by  trying  to  make  as 
accurate  predictions  as  possible. 

•  Control  is  the  attitude  that  one  should  attempt  to  minimise  uncertainty  by  seizing  the  initiative  and 
taking  control  of  the  relevant  bits  of  the  environment. 

•  Acceptance  is  the  attitude  that  one  can  never  know  what  is  going  to  happen,  so  there  is  no  point  in 
trying  to  predict  or  reduce  uncertainty  in  any  way. 


Control 


Predict  Accept 


Figure  2-2:  The  So-Called  Uncertainty  Triangle,  Denoting  Basic 
Attitudes  Towards  Uncertainty  (Extended  from  [7]). 


Any  of  the  three  attitudes  may  lead  to  some  form  of  preparation.  This  could  range  from  focusing  on  those 
elements  of  uncertainty  that  can  be  predicted  and  design  options  accordingly,  to  designing,  for  example, 
a  flexible,  module-based  response  strategy  in  preparation  of  any  contingency  or  challenge  that  might 
occur. 

The  idea  behind  the  triangle  is  that  a  client  should  be  aware  of  these  attitudes,  and  that  a  proper  mix  of  all 
three  will  be  the  most  fruitful  approach  to  preparing  for  uncertain  events.  For  a  messy  problem,  adopting 
only  one  of  these  attitudes  is  rarely  sufficient. 

It  should  be  clear  that  it  will  not  be  sufficient  to  address  uncertainty  by  merely  varying  a  few  of  the 
parameters  at  the  end  of  the  study  as  part  of  a  sensitivity  analysis.  The  attitude  towards  uncertainty  will 
shape  the  design  of  the  study,  specifically  its  problem  framing  stage  and  the  stage  where  the  type  of 
options  should  be  considered  [7]. 
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In  subsequent  chapters  specific  aspects  of  uncertainty  will  be  discussed  in  more  depth.  For  example, 
how  the  nature  of  the  uncertainty  affects  the  design  of  the  study’s  (‘soft’)  methodology  is  discussed  in 
Chapter  6. 


2.4  CHARACTERISTICS  OF  HARD’  AND  ‘SOFT’  OA 

In  the  academic  literature  there  has  been  some  debate  about  the  relationship  between  ‘hard’  and  ‘soft’  OA. 
Some  argue  that  they  are  two  different  ways  of  thinking,  even  two  radically  different  paradigms5 
considering  the  differences  in  problematic  situations  for  which  they  are  most  appropriate  and  the  different 
methods  they  encompass.  The  CoBP  adopts  the  view  of  others  who  argue  that  the  two  are  complementary 
and  can  be  used  in  series  or  in  parallel,  and  even  that  ‘hard’  methods  can  be  accommodated  within  a  larger 
framework  of  ‘soft’  OA  methods  and  vice  versa.  Even  with  ‘puzzles’  to  be  solved  using  some  ‘hard’  OA 
technique  there  will  always,  at  some  level  of  the  hierarchy  of  clients  or  at  some  stage  of  the  study  process, 
be  different  interpretations  of  data  or  results  or  validation  findings.  There  will  also  be  experts  providing 
estimates  based  on  judgement;  and  there  will  be  group  discussions  in  order  to  arrive  at  a  consensus  about, 
for  example,  the  problem  formulation  to  begin  with.  All  these  matters,  also  to  be  addressed  in  a  ‘hard’  OA 
study,  require  approaches  that  are  based  on  judgement  and  thus  are  related  or  even  belong  to  the  area 
of  ‘soft’  OA.  In  addition,  a  ‘soft’  OA  approach  may  turn  to  the  use  of  some  ‘hard’  method  to  solve  a 
well-defined  problem  as  part  of  a  messy  problematic  situation6.  This  will  be  shown  in  the  illustrative  study 
scenario  depicted  in  Figure  3-1  in  Chapter  3. 

In  previous  sections  a  large  number  of  aspects  of  ‘soft’  and  ‘hard’  OA  have  been  addressed.  They  pertain 
to  the  nature  of  the  problematic  situation  for  which  they  are  most  suited,  the  purpose  and  outcome  of  a 
study,  the  people  involved,  etc.  In  the  literature  numerous  taxonomies  and  tables  can  be  found  highlighting 
similarities  and  differences  between  the  two  types  of  OA.  The  most  useful  distinctions  are  summarised 
below  in  Table  2-2  instead  of  providing  formal  definitions  of  ‘soft’  and  ‘hard’  OA.  The  descriptions  tend 
to  emphasise  the  differences  and  sometimes  indicate  extremes,  but  one  should  be  aware  that  each  aspect 
listed  in  Table  2-2  represents  a  spectrum  of  possibilities. 


5  A  ‘ paradigm  ’  is  a  set  of  philosophical  assumptions  about  what  should  be  done  and  how  (and  the  ethics  of  it)  and  about  the 
nature  of  existing  knowledge  (and  its  validity)  (based  on  [8]). 

6  As  a  further  clarification:  one  of  the  methodologies  that  is  often  applied  in  a  ‘soft’  OA  is  ‘Soft  Systems  Methodology’  [9], 
The  adjective  ‘soft’  in  SSM  however  refers  to  the  way  of  dealing  with  problematic  situations,  not  the  nature  of  that  situation 
itself.  SSM  can  assist  in  addressing  puzzles  as  well. 
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Table  2-2:  Differences  Between  ‘Hard’  and  ‘Soft’  OA  as 
Extremes  of  a  Spectrum  (Based  on  e.g.  [1],  [3],  [10]). 


Aspects  and 

Elements  of  the  Study 

‘Hard’  OA 

‘Soft’  OA 

Methodology 

Based  on  generally  accepted  and  theoretically 
grounded  views  of  analysis. 

Predominantly  factual  and  objective. 

Based  on  thoroughly  discussed  arguments 
accepted  by  study  participants. 

Predominantly  interpretative,  subjective, 
inductive. 

Proven  value  by  experience. 

Models 

Shared  representation  of  problematic  part  of 
the  real  world. 

Unambiguously  linked  to  the  study  purpose 
and  assumed  to  produce  a  problem  solution. 

Utilise  often  (in  addition  to  ‘soft’ 
representations)  a  formulaic  representation  of 
relationships  between  quantified  concepts. 

Representation  of  problematic  part  of  the 
perceived  world  based  on  and  shared  by  some 
or  all  study  participants. 

Help  understand  study  purpose  and  contribute 
to  improving  problem  issues. 

Utilise  the  power  of  visualisation  of  concepts 
and  their  relationships  often  using  well-chosen 
verbs  and  nouns. 

Methods 

Predominantly  mathematics-based. 

Algorithmic. 

Predominantly  judgement-based,  governed  by 
guidelines  and  non-mathematical  rules. 

Data 

Based  on  observer-independent 
measurement. 

Uncertainty  treated  stochastically. 

Based  usually  on  observer-dependent 
judgement. 

Uncertainty  mostly  treated  qualitatively. 

Study  Outcomes 

Based  on  rational,  quantitative  analysis. 

Based  on  clear  results. 

Nature:  ‘solutions’.  Search  for  the  optimum. 

Repeatable. 

Based  on  explorative,  qualitative  analysis. 

Based  on  insights  and  learning. 

Nature:  ‘ways  forward’.  Search  for  the  requisite 
and  accepted. 

Often  not  repeatable. 

Study  Purpose 

Based  on  problem  analysis.  Clear  at  the 
outset  and  then  taken  as  given. 

Can  become  clear  after  problem  analysis  or 
even  after  one  or  more  study  stages. 

Problematic  Situation 

Problematic  situation  is  abstraction  of  the  real 
world  and  can  be  delineated  and  engineered 
as  a  ‘system’  with  its  components. 

Usually  short-term,  operations-oriented. 

Single  objective.  Multiple  objectives  and 
perspectives  can  be  modelled. 

Problematic  situation  is  often  a  mental  construct 
and  cannot  easily  or  not  at  all  delineated. 
Assembly  of  (partly)  related  ‘concepts’. 

Usually  long-term,  strategy-oriented. 

Multiple  objectives  and  perspectives  exist, 
usually  based  on  different  stakeholder  values 
that  are  difficult  to  reconcile. 

Process  - 
Study  Stages 

Can  be  designed  ex-ante.  Likely  to  be 
sequential. 

Ex-ante  design  in  general  terms.  Stages 
emerge  as  appropriate.  Likely  to  be  iterative. 

Process  - 
People  Involved 

Clients  (and  other  stakeholders)  provide  input 
to  problem  analysis  and  model  formulation. 

Power  and  emotion  do  not  really  matter. 

Analysts  suggest  and  conduct  the  formal 
analysis,  avoid  biases,  present  solutions. 

Clients  (and  other  stakeholders)  participate  fully 
in  process  and  can  appear  in  a  model-based 
representation  of  a  problematic  situation. 

Power  and  emotion  matter. 

Analysts  suggest  and  facilitate  the  process  and 
any  model  usage,  accommodate  biases  from 
different  perspectives,  engage  clients  in  actively 
identifying  options. 

Although  Table  2-2  suggests  differences  between  the  two  approaches,  the  common  denominator  is  the 
general  posture  of  decision  support. 
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The  importance  of  Table  2-2  already  arises  at  an  initial  stage  of  a  study  when  the  analyst  has  to  negotiate 
the  study  approach  with  the  client.  Initial  questioning  based  on  the  entries  of  that  table  will  provide  the 
analyst  with  information  regarding  for  example  (see  Chapter  6  for  further  discussion): 

•  The  nature  of  the  problematic  situation  and  the  reason  why  it  should  be  resolved; 

•  The  people  (or  types  of  expertise)  to  be  involved; 

•  The  amount  of  effort  required  in  getting  a  clear  and  shared  picture  among  client  and  other 
stakeholders  about  what  is  to  be  achieved  by  the  study,  under  what  conditions  an  achievement  is 
regarded  as  a  success,  and  the  nature  of  the  information  they  can  provide; 

•  The  nature  of  the  methodology/method  to  use  (or  even  a  combination  of  methodologies  and 
methods),  and  the  nature  of  data  required; 

•  The  study  phasing;  and 

•  The  general  attitude  of  the  analyst  and  the  client:  a  traditional  expert  mode  (where  the  client  can  rely 
on  the  analyst  to  solve  the  problem),  or  a  facilitated  mode  (where  analyst  and  client/stakeholders 
work  together  to  create  satisficing  options)  [3]. 

Table  2-2  leads  to  four  positive  elements  for  a  judgement-based  approach  over  a  purely  quantitative  one: 

•  The  nature  of  the  models  involved  lead  to  more  of  a  conceptual  approach,  allowing  more  freedom 
in  addressing  the  essence  of  the  issue  and  thus  avoiding,  possibly  unconsciously,  adherence  to  the 
original  paradigm; 

•  A  creative  use  of  uncertainty,  allowing  ‘what  if  considerations; 

•  A  recognition  of  the  need  to  consider  all  sides  of  the  issue;  and 

•  An  appreciation  of  how  to  handle  different  viewpoints  and  belief  systems. 

This  implied  creativity  leads  to  a  divergent  discovery  phase  where  all  aspects  of  the  issue  are  discussed  and 
explored.  This  is  where  completely  new  options  or  approaches  may  be  discovered.  This  can  be  compared 
with  the  case  for  more  structured  issues  (e.g.  ‘hard’  OA)  where  there  will  be  steady  convergence  from  the 
start.  The  issue  of  divergence  and  convergence  will  be  addressed  again  in  Chapter  5  (Figures  5-8  and  5-9). 

2.5  THE  NATURE  OF  DATA  AND  MODELS  IN  SOFT’  AND  ‘HARD’  OA 

It  may  be  useful  to  focus  on  two  prominent  elements  of  an  operational  analysis:  the  data  and  the  model 
and  method  for  analysis.  This  discussion  will  not  exactly  define  ‘hard’  and  ‘soft’  OA  more  precisely  than 
the  previous  section,  which  would  be  a  daunting  task,  but  may  create  a  deeper  understanding  of  the  nature 
of  both. 

Figure  2-3  shows  a  scheme  of  different  instances  of  the  nature  of  data  and  models  and  methods  and  their 
combinations.  In  italics,  some  examples  are  offered  to  illustrate  the  general  descriptions.  Figure  2-3a 
shows  the  shaded  parts  of  the  scheme  that  are  typical  of  ‘hard’  OA;  Figure  2-3b  shows  the  ‘soft’  OA  parts. 
The  combination  of  ‘subjective’  and  ‘quantitative’  appears  shaded  in  both.  It  depends  on  where  one  puts 
the  emphasis:  the  elicitation  of  this  type  of  data  may  use  ‘soft’  methods,  the  result  however  is  treated  as 
input  to  a  quantifiable,  mathematical  model,  although  some  uncertainty  may  be  involved.  Even  the  degree 
of  uncertainty,  for  example  when  expressed  as  a  quantitative  interval,  may  be  used  as  numerical  input  to  a 
mathematical  model,  although  the  elicitation  of  that  interval  is,  again,  essentially  a  judgement-based 
process7.  All  this  illustrates  that  the  distinction  between  ‘hard’  and  ‘soft’  is  not  as  sharply  delineated  as 


7  Another  example  would  be  the  elicitation  of  input  data  for  the  Analytic  Hierarchy  Process  (a  decision-making  methodology 
based  on  a  hierarchy  of  multiple  criteria,  [11])  which  requires  a  judgement-based  elicitation  process.  However,  the  judgement 
data  are  processed  by  mathematical  means  to  yield  relative  frequencies  in  the  hierarchy. 
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suggested  by  the  boxes  in  Figure  2-3;  the  boundaries  between  the  boxes  are  rather  blurred  in  practice  and 
the  arrows  indicate  that  the  scales  are  floating. 


Objective 

I 

Subjective 


Data  Method/Model 


Description-/ 
observation  -based; 
ordinal,  nominal; 
agreed  as  ‘true’. 

The  Eiffel  Tower  is  higher 
than  the  Reichstag 

Measurement-based; 
numbers; 
agreed  as  ‘true'. 

The  Eiffel  Tower  is 

300.51  m  high. 

Mathematically 
expressed  relations  and 
aggregation/search  rules; 
agreed  as  True’. 

What  is  minimum  number 
of  staff  required? 

(->  Wcighzcd  Sums.  Linear 
Programming,  Queuing 
Theory } 

Opinion-based; 

Opinion-based; 

Non-mathematically 

ordinal,  nominal; 

numbers; 

expressed  relations  and 

debatable. 

debatable. 

aggregation/search  rules; 
debatable  requisite 

How  can  Eiffel  Tower  be 

The  Eiffel  Tower  is 

The  weight  of  the  Eiffel 

made  more  attractive ? 

prettier  than  the 

Tower  is  estimated  at 

( -9  Cognitive  Maps. 

Reichstag 

app.  10.000  tonnes. 

Morphological  Box) 

Qualitative  ^ 

^  Quantitative 

Figure  2-3a:  Combinations  of  Data  and  Method/Model 
Types  in  ‘Hard’  OA  (Shaded),  with  Examples. 


Data 


Method/Model 


Objective 

I 

Subjective 


Description -/ 
observation  -based; 
ordinal,  nominal; 
agreed  as  true’. 

The  Eiffel  Tower  is  highei 
than  the  Reichstag. 

Measurement-based: 

numbers; 

agreed  as  true’. 

The  Eiffel  Tower  Is 

300.51  m  high 

Mathematically 
expressed  relations  and 
aggregation/search  rules; 
agreed  as  ‘true’. 

W hat  is  minimum  number 
of  staff  required? 

( Weighted  Sums.  Linear 
Programming.  Queuing 
^Theonfl^ 

Opinion -based; 

Opinion-based; 

tion-mathem  ati  c  a  lly 

ordinal,  nominal; 

numbers; 

expressed  relations  and 

debatable 

debatable. 

aggregation/search  rules; 

debatable,  requisite. 

Wow  can  Eiffel  Tower  be 

The  Eiffel 

Tower  is 

The  weight  of  the  Eiffet 

made  more  attractive ? 

prettier  than  the 

Tower  is  estimated  at 

{->  Cognitive  Maps. 

\  Reichstag 

app.  10.000  tonnes. 

Morphological  Box) 

Qualitative 


Quantitative 


Figure  2-3b:  Combinations  of  Data  and  Method/Model 
Types  in  ‘Soft’  OA  (Shaded),  with  Examples. 
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Model-based  decision  support  is  the  primary  feature  of  OA  and  so  it  is  of  ‘soft’  OA.  The  type  of  models 
built  and  used  (by  using  appropriate  methods)  in  ‘soft’  OA,  however,  is  different  from  that  of  traditional 
OA,  as  discussed  in  Section  2.4.  The  reason  for  this  is  that  the  nature  of  the  problematic  situation  addressed  by 
‘soft’  OA  is  significantly  different. 

In  general,  building  a  model  will  enable  (based  on  [3]): 

•  A  convenient  organisation  (and  most  often  visualisation)  of  the  defining  elements  of  the  problematic 
situation  and  their  interrelationships; 

•  A  representation  of  a  large  amount  of  (qualitative  and  quantitative)  information; 

•  Gaining  a  better  understanding  of  the  way  in  which  factors  influencing  the  problematic  situation 
interact  with  the  constitutive  elements  of  that  situation; 

•  Designing  and  investigating  decision  options  and  their  consequences  and  determining  which 
option(s)  might  be  preferred  over  other  options;  and 

•  Enabling  stakeholders  to  learn  about  the  nature  and  root  causes  of  a  problematic  situation  by 
‘playing’  with  the  model  and  thereby  also  gaining  a  better  understanding  of  each  other’s  viewpoints 
(i.e.  the  model  is  a  learning  tool). 

A  ‘soft’  OA  model  is,  above  all,  a  means  to  help  people  think  about  a  problematic  situation,  understand 
and  interpret  its  elements  and  factors  of  influence  and  their  interrelationships,  express  and  deal  with  views 
and  opinions,  identify  crucial  issues  and  come  up  with  ways  forward  to  address  and  cope  with  them. 

By  contrast,  a  ‘hard’  OA  model  is,  above  all,  a  means  to  search  for  an  optimal  solution  to  a  well-defined 
problematic  situation,  or  a  ‘satisficing’  solution  using  mathematics-based  heuristic  algorithms. 


2.6  BENEFITS  AND  OPPORTUNITIES  OF  DOING  ‘SOFT’  OA 

Now  that  the  term  ‘soft’  OA  has  been  introduced  and  some  key  concepts  and  issues  have  been  discussed, 
and  before  the  guidelines  for  conducting  a  ‘soft’  OA  study  will  be  discussed  in  subsequent  chapters, 
this  chapter  concludes  by  asking  why  one  would  use  ‘soft’  OA  in  the  first  place. 

There  is  a  strong  similarity  between  judgemental  methods  in  operational  research  practice  and  a  number  of 
other  modes  of  professional  analytical  practice.  The  closest  relationship  is  perhaps  with  the  domains  of 
evaluation  [12],  social  science  (as  described  by  SAS-074  [13]),  business  analysis,  and  elicitation  (from 
experts)  [14], 

There  are  many  issues  when  using  traditional  analysis  techniques  to  analyse  complex  problems, 
as  discussed  earlier  (e.g.  in  particular,  understanding  a  system  fully  will  assist  in  understanding  the  future 
behaviours  of  such  a  system).  ‘Soft’  OA  can  incorporate  multiple  perspectives  of  the  system  and/or  the 
problem  and  can  help  define  boundaries  and  clarify  assumptions  and  stakeholders’  mental  models. 

‘Soft’  OA  techniques  can  be  used  to  try  to  make  sense  of  these  issues  by  bringing  multiple  stakeholders 
together  to  consider  their  multiple,  often  conflicting,  views  of  the  problem  space.  In  particular,  where  there 
is  a  lack  of  clarity  about  the  problem  definition,  disagreement  and  uncertainty  about  the  issues  and 
decision  making,  ‘soft’  OA  techniques  can  play  a  large  role  in  enabling  stakeholders  to  make  sense  of  the 
situation. 

‘Soft’  OA  techniques  can  help  to  manage  the  complexity  within  a  problem  space.  They  can  consider  both 
qualitative  and  quantitative  data  and  information,  including  data  that  is  incomplete  or  conflicting.  Indeed 
‘soft’  OA  techniques  are  based  on  the  judgement  of  stakeholders  who  are  involved  in  the  decision  making. 
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For  this  reason,  ‘soft’  OA  techniques  tend  to  be  subjective  in  nature  and  hence  multiple,  and  most  likely 
conflicting,  opinions  of  stakeholders  could  exist.  But  judgement  can  be  treated  as  objectively  as  possible 
by  following  rules  grounded  in  theory  or  established  practice  and  by  recording  argumentation  and  rationale 
and  agreeing  on  a  sound  process  accepted  by  all  involved. 

In  this  remit,  ‘soft’  OA  can  help  to  conceptualise  and  visualise  the  problem  space  and  the  issues,  constraints 
and  uncertainties  within  it.  It  starts  to  provide  some  structure  to  the  problem  to  obtain  a  common 
understanding  of  the  issues  across  stakeholders. 

When  using  ‘soft’  OA  techniques,  there  is  often  the  need  for  a  coherent  and  collaborative  problem  solving 
process  because  of  the  varied  and  conflicting  stakeholder  objectives  and  worldviews.  A  ‘soft’  OA  analyst 
also  needs  to  play  the  role  of  facilitator  in  order  to  achieve  this  successfully.  Hence  there  is  often  the  need 
for  workshops  and  interviews  in  order  to  formally  gather  information.  It  is  for  this  reason  that  ‘soft’ 
methods  typically  use  subjective  knowledge  based  on  the  judgement  of  stakeholders  who  are  familiar  with 
the  problem  space. 

Stakeholders  may  have  different  priorities  and  different  ways  of  working.  ‘Soft’  OA  can  start  to  explore 
the  disagreements  and  uncertainties  that  exist  so  that  an  agreed  consensus  can  be  developed  and  action  can 
be  taken.  ‘Soft’  methods  also  enable  individuals  to  consider  other’s  points  of  view,  and  to  consider  how 
other  people  interpret  the  same  experiences.  When  using  these  methods  one  can  start  to  map  out  the 
relationships  between  individuals  involved. 

‘Soft’  OA  can  help  one  to  start  to  make  sense  of  the  key  issues  and,  although  it  often  does  not  produce  a 
perfect  solution  (there  is  no  optimal  solution  to  a  messy  problem),  it  can  be  a  rather  good  approach  to 
obtain  a  common  understanding  of  the  problem  and  the  decision  space,  and  to  gain  consensus  amongst 
stakeholders  on  the  way  ahead. 
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•  Chapter  1  •  INTRODUCTION 

•  Chapter  2-  PROBLEMATIC  SITUATIONS  AND  ‘SOFT’  OA 

•  Chapter  3 -ACHIEVING  VALIDITY,  CREDIBILITY  AND  ACCEPTANCE 


Chapter  4  ■  ROLES  AND  RESPONSIBILITIES 


•  Chapter  5  ■  THE  ‘SOFT’  OA  STUDY  PROCESS 

•  Chapter6-  DESIGNING THE'SOFT’OA  STUDY  METHODOLOGY 

•  Chapter  7  -  DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 

•  Chapter  8-  INTERPRETING,  USING  AND  COMMUNICATING  RESULTS  AND  OUTCOMES 


•  Chapter  9  ■  THE  ESSENCE  OF  THE  CODE  OF  BEST  PRACTICE 

•  Annex  A  -  LIST  OF  KEY  STATEMENTS  AND  RECOMMENDATIONS  OF  IMPORTANCE 

TO  THE  ANALYST 

•  Annex  B  •  SELECTIVE  LI  ST  OF  'SOFT  METHODS 


>  The  CoBP  identifies  how  validity,  credibility  and  acceptance  can  be  achieved  in  a  ‘soft’  OA  study. 

>  The  primary  dimensions  of  validity  are  objectivity  and  rigour;  credibility  and  acceptance  are  not 
distinct  qualities  but  are  rather  derived  from  validity. 

>  The  analyst’s  aim  should  be  to  make  a  clear  distinction  between  the  reality  which  is  shared  amongst 
the  stakeholders,  and  the  sectional  perspectives  which  each  may  propagate  for  his  own  reasons. 

>  Gathering  subjective  judgements  from  experts  exposes  the  analyst  and  his  study  to  bias  which  can 
be  motivational  or  cognitive  in  nature. 

>  The  biggest  threat  to  validity  is  ignorance  at  the  design  stage  of  what  is  likely  to  be  important.  This 
threat  can  itself  be  mitigated  by  creating  an  iterative  design  similar  to  that  used  in  experimentation. 

>  An  analyst  needs  to  conduct  activities  in  an  ethical  manner  that  deserves  the  confidence  of  all 
parties  involved. 

>  Credibility  and  acceptance  will  be  reinforced  by  independent  scrutiny. 
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3.1  INTRODUCTION 

The  previous  chapters  of  this  Code  have  set  out  the  aims  and  purposes  of  a  Code  of  Best  Practice  and 
discussed  the  characteristics  of  ‘soft’  OA  and  judgement-based  analysis.  This  chapter  will  describe  the 
general  context  and  process  in  which  judgement-based  methods  are  applied,  and  show  where  and  why  issues 
of  validity  have  impact  on  the  utility  of  the  methods.  The  primary  aspect  of  relevance  to  stakeholders  in  the 
use  of  these  methods  is  validity.  Credibility  and  acceptance  of  a  method  are  not  separable  from  validity  but 
are  rather  derived  from  it.  This  chapter  will  discuss  the  nature  of  all  three  concepts  and  conclude  with  a 
discussion  of  ethical  aspects  of  conducting  a  (‘soft’)  OA  study. 


3.2  THE  CONCEPTUAL  ENVIRONMENT  OF  ‘SOFT’  METHODS 

The  typical  conceptual  environment  in  which  ‘soft’  methods  come  into  play  is  shown  in  Figure  3- 1 1 . 
Decision  makers,  faced  with  a  need  to  formulate  a  plan  of  action,  will  express  their  predicament  as  a 
problematic  situation  to  which  some  structure  should  be  given.  Analysts  are  asked  to  suggest  a  design  for  a 
study  of  the  problematic  situation;  the  design  evokes  methods,  models  and  data  in  an  iterative,  and  hopefully 
convergent,  programme  of  analysis  which  may  include  objective  knowledge  from  the  worlds  of  science, 
mathematics  and  engineering. 


1  This  figure  in  itself  is  an  example  of  ‘soft’  OA:  a  concept  map  where  key  concepts  and  their  relationships  are  depicted  in  order 
to  create  a  structured  visual  image  of  the  ‘problematic  situation’.  Its  purpose  is  creating  clarity,  focus  and  enabling 
communication  and  debate. 
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Figure  3-1 :  ‘Soft’  OA  and  the  Role  of  the  CoBP. 


The  CoBP  is  written  with  this  environment  in  mind.  Chapter  2  has  explained  the  difficulties  of  defining 
complex  problematic  situations,  discussed  aspects  that  characterise  them  and  related  problematic  situation 
types  with  ‘soft’  and  ‘hard’  OA.  It  also  explains  why  ‘soft’  OA  can  be  helpful  to  a  decision  maker.  Chapter  4 
explains  the  roles  and  responsibilities  of  the  clients,  analysts,  facilitators,  and  other  stakeholders  within  a 
typical  ‘soft’  OA  study.  Chapters  5,  6,  and  7  discuss  the  action  plans  which  will  deal  with  complex 
problematic  situations  and  detail  the  model  building  and  analysis  process  which  in  the  above  diagram  is 
represented  as  an  iterative  process  combining  stakeholder  views,  data  collection,  and  analyst  expertise  in 
choosing  the  appropriate  modelling  approach.  Chapter  8  of  this  volume  explains  how  to  engage  the  client 
(and  other  stakeholders)  to  best  communicate  results  to  them,  ensuring  the  study  outcomes  are  of  most  value. 
The  concepts  of  validity,  credibility  and  acceptance  are  incorporated  into  all  aspects  of  this  conceptual 
environment  and  each  links  naturally  into  the  study  design  process. 
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Analysts  should  test  their  study  design  against  this  CoBP  in  order  to  establish  the  validity  and  credibility 
of  their  approach.  The  structured  understanding  which  flows  from  the  analysis  forms  the  basis  of  insights 
and  advice  on  the  proposed  courses  of  action  to  the  decision  makers. 

This  process  is  a  re-expression  in  the  decision-making  context  of  the  general  approach  to  systematic 
enquiry  and  assessment.  It  therefore  raises  similar  philosophical  questions  relevant  to  the  issues  of  validity 
and  credibility. 

3.3  THE  PHILOSOPHY  OF  METHODS 

This  is  an  extensive  field  which  this  CoBP  cannot  hope  to  address  completely.  However,  it  is  important  to 
understand  the  foundation  (‘grounding’)  of  methods  so  that  judgement-based  approaches  can  be  seen  in 
relation  to  the  conventional  methods  of  science,  engineering  and  mathematics. 

Consideration  of  the  philosophical  foundations  has  more  than  general  relevance;  it  may  also  be  helpful  in 
the  conduct  of  individual  studies  in  so  far  as  the  study  domain  is  an  instance  of  this  general  philosophical 
context. 

The  world  view  of  the  researcher  can  be 
expressed  in  terms  of  the  three  aspects  to  the 
philosophy  of  knowledge  and  the  methods  use 
to  gain  it:  ontology,  epistemology,  and 
axiology  (see  also  [1]): 

•  Ontology  -  What  are  the  elements  (constituents)  of  discourse  and  consideration,  and  do  they 
endure?  Are  the  elements  objective  (‘real’  or  ‘true’;  see  also  Section  3.4.1)  or  are  they  constructed 
and  so  dependent  for  their  meaning  and  interpretation  on  the  cultural  place  in  which  they  are  posited? 

•  Epistemology  -  What  meaning  is  to  be  drawn  from  a  set  of  the  elements  and  their  relations? 
Are  the  meanings  and  relations  objective  (‘real’)  and  time  independent,  or  are  they  (also) 
constructed  and  context-dependent?  What  enduring  principles  can  be  expressed  in  connection 
with  the  ontologically  derived  elements  and  can  we,  through  reflection,  reason  about  them? 
Such  considerations  bear  upon  the  power  of  explanation  which  a  method  gives  its  users.  It  may 
also  have  effect  through  its  aesthetic  qualities,  a  more  satisfying  construct  leading  to  enhanced 
explanatory  power.  Such  considerations  stray  into  the  third  aspect,  axiology. 

•  Axiology  -  What  is  held  to  be  worthy  or  of  value?  Axiology  is  concerned  with  the  (ultimate) 
purpose  of  the  research.  Two  perspectives  are  relevant:  that  of  knowledge  for  its  own  sake, 
and  that  of  knowledge  which  informs  action.  In  practice,  it  may  not  be  possible  to  decide  in  which 
category  specific  knowledge  resides.  Much,  if  not  all,  of  the  work  conducted  by  the  users  of  this 
CoBP  will  serve  the  second  category  of  knowledge  (i.e.  which  informs  action).  The  worth  or 
value  of  analysis  practice  and  its  products  is  also  strongly  dependent  on  the  application  of 
appropriate  ethical  principles.  In  particular,  the  general  principles  of  good  scientific  practice, 
an  analyst  and  client’s  honesty  and  openness,  and  the  rigour  of  the  study  process  are  key 
considerations.  Ethical  considerations  are  discussed  in  more  depth  later  in  this  chapter. 

Each  practitioner  needs  to  establish  his  position  in  respect  of  these  three  philosophical  aspects.  Many  find 
that  a  pragmatic,  critical  realist  approach2  is  their  natural  position.  A  critical  realist  holds  that  knowledge  is 
gained  from  the  real  world  through  the  use  of  our  senses  assisted  by  our  critical  powers  of  reasoning. 
Knowledge  can  be  both  captured  objectively  by  measurement,  and  constructed  subjectively  from  the 


What  is  the  grounding  of  the  techniques  used  in 
judgement-based  analysis?  A  good  analyst  should  always 
understand  the  philosophical  foundation  of  his  tools  and 
methods. 


2 

Implying  also  the  capture  of  authentic  knowledge,  based  on  sense  experience  and  empirical  verification.  A  critical  realist 
holds  that  science  should  be  understood  as  an  ongoing  process  in  which  scientists  improve  the  concepts  they  use  to  understand 
the  mechanisms  that  they  study  (Wikipedia).  A  further  discussion  of  this  subject  can  be  found  in  Chapter  7. 


3-4 


RTO-TR-SAS-087 


ACHIEVING  VALIDITY,  CREDIBILITY  AND  ACCEPTANCE 


expectations,  world  view,  and  values  of  the  stakeholders  in  a  decision  situation.  In  essence,  the  aim  of  the 
critical  realist  should  be  to  make  a  clear  distinction  between  the  reality  which  is  shared  amongst  the 
stakeholders,  and  the  sectional  perspectives  which  each  may  propagate  for  his  own  reasons.  Wide  experience 
has  shown  this  position  to  be  feasible,  not  least  because  it  is  universally  recognised  by  stakeholders. 


3.4  THE  NATURE  OF  VALIDITY 
3.4.1  Dimensions  of  Validity 

The  primary  dimensions  of  validity  are  objectivity  and  rigour,  as  defined  by  the  Concise  Oxford  English 
Dictionary  (as  below).  In  essence,  objectivity  refers  to  the  ontological  aspect  of  a  method  and  its  data, 
whilst  rigour  refers  to  their  essential  epistemological  quality.  Supplementary  aspects  of  validity  which 
reflect  axiological  considerations  and  so  may  be  present  in  differing  intensity  according  to  the  context  in 
which  a  method  is  applied,  include: 

•  Repeatability; 

•  Auditability/transparency; 

•  Independence  and  lack  of  bias; 

•  Grounding/consistency  (in/with  standards,  and  other  data  and  methods); 

•  Understand-ability; 

•  Explanatory  power; 

•  Completeness; 

•  Robustness  under  uncertainty;  and 

•  Clear  separation  of  data  from  the  method  which  uses  it. 

The  dimensions  of  validity  can  be  accounted  as  follows. 

•  Objectivity.  Analysis  should  be  objective  -  i.e.  based  on  rules  grounded  in  theory  or  established 
practice  and  characterised  by  recorded  argumentation  and  rationale  and  following  an  agreed  and 
sound  process  accepted  by  all  involved.  A  good  test  of  objectivity  is  whether  an  analysis  is  capable 
of  debate  amongst  the  stakeholders. 

•  Rigour  is  achieved  through  strict  enforcement  of  logical  rules  and  doctrine,  such  as  this  CoBP. 
Methods  should  be  applied  as  proposed  and  documented,  and  any  deviations  should  be  justified 
and  documented. 

•  Repeatability  -  sometimes  called  reproducibility  or  replicability  -  is  the  quality  of  a  phenomenon 
to  occur  again,  possibly  in  different  places  and  times,  and  observed  by  different  people.  It  is  a  key 
characteristic  of  the  scientific  study  of  phenomena,  and  may  be  difficult  to  achieve  when  (partly) 
relying  on  judgement  as  some  conditions  of  the  study  domain  (not  least  the  expert  stakeholders) 
may  be  beyond  the  control  of  the  researcher.  However,  it  should  be  regarded  as  a  worthwhile 
objective.  It  is  often  the  justification  for  the  re-iteration  of  studies  whose  results  have  been 
challenged. 

•  Auditability /Transparency.  Analysis  should  be  trustworthy,  with  no  undeclared  assumptions  or 
unstated  simplifications  (see  rigour,  above).  Any  given  method  needs  to  be  both  verified  and 
validated.  Verification  tests  whether  the  method  works  as  it  is  specified  to  do,  whilst  validation 
results  from  a  (satisfactory)  comparison  with  a  standard  of  behaviour  external  to  the  application  to 
hand.  Tests  of  verification  and  of  validation,  taken  together,  may  be  referred  to  as  the  execution  of 
due  diligence.  Analysts  have  an  obligation  to  monitor  and  report  their  own  analytical  processes, 
including  those  taken  to  establish  the  validation  state  of  a  method,  e.g.  through  the  use  of  method 
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logbooks.  Transparency  to  enable  repeatability  may  not  be  the  only  goal;  transparency  to 
demonstrate  the  legitimacy  of  the  process  undertaken  is  just  as  important.  The  assumptions  and 
mental  processes  that  underpin  the  design  must  be  documented  alongside  the  results  so  that  they 
can  be  scrutinised  together.  Political  and  power  factors  may  limit  the  transparency  achievable  in 
judgement-based  OA;  this  is  discussed  further  in  Chapters  4,  6  and  7. 

•  Independence  and  Lack  of  Bias.  For  some,  the  analysis  should  be  independent  of  vested  interests 
[2].  For  others,  vested  interests  are  part  of  the  analysis  and  should  be  captured.  The  analyst  must 
reflect  on  and  deal  carefully  with  bias,  where  possible  mitigating  or  capturing  it.  Perceived 
objectivity  of  the  analyst  may  be  questioned.  The  relationship  of  the  analyst  with  the  customers, 
stakeholders  and  participants  will  be  a  factor.  The  role  of  the  analyst  within  the  method  should 
form  an  explicit  element  of  the  study  design.  The  analyst  should  regard  himself  as  accountable  for 
the  quality  of  the  analytical  work;  he  needs  to  address  arising  issues  with  integrity,  and  in  a 
neutral  and  impartial  manner.  Awareness  of  the  political  and  social  norms,  values  and  power 
structure  of  the  stakeholders  is  important  in  determining  the  perceived  independence  of  the 
method  and  its  analyst  executors,  and  thus  the  entire  validity  of  the  design  of  a  judgement-based 
study.  Political  issues  in  particular  might  be  very  difficult  to  accommodate  into  a  study  which 
seeks  to  meet  criteria  of  validity.  Issues  of  independence  are  discussed  further  in  Chapter  4. 

•  Grounding/Consistency  (in/with  standards,  and  other  data  and  methods).  Methods  should  be  based 
on  solid  theoretical  and  philosophical  grounds.  Also,  where  appropriate,  they  should  be  validated 
comparatively  against  real  data  gathered  in  designed  experiments. 

•  Understand-ability .  Questions  to  be  addressed  include:  is  the  nature  and  methods  of  the  enquiry  or 
study  fully  understood  by  all  stakeholders?  Is  the  need  for  validation  well  understood  and  is  the 
nature  of  the  processes  of  validation  inclusive?  What  is  the  impact  of  judgement  biases  and 
‘heuristics’,  as  discussed  below? 

•  Explanatory  Power.  What  depth  and  breadth  of  explanation  of  the  system  properties  and  behaviours 
which  the  problematic  situation  embodies  is  afforded  by  the  OA  method  and  its  results? 

•  Completeness.  Do  the  methods  adopted  for  the  study  address  all  aspects  of  a  problematic 
situation,  and,  if  not,  in  what  sense  does  their  omission  impact  the  study? 

•  Robustness  under  Uncertainty.  The  analysis  should  acknowledge  uncertainty  in  data  and  method 
right  from  the  outset,  and  a  plan  should  be  formed  to  reduce  the  impact  of  uncertainty  and  manage 
the  residue  so  that  the  outcome  of  the  analysis  can  be  said  to  be  robust,  i.e.  of  known  sensitivity  to 
variations  in  the  inputs,  both  structural  and  parametric.  Issues  of  uncertainty  are  discussed  more 
in-depth  in  Chapters  2  and  6. 

•  Separation  of  Data  from  the  Method  which  Uses  it.  The  overall  methodology  should  strive  to 
make  a  clear  distinction  between  the  methods  adopted  by  the  analysis,  and  the  data  those  methods 
will  employ.  The  distinction  need  not  be  fixed  for  the  entire  period  of  analysis;  in  general,  even  a 
method  could  be  said  to  be  a  form  of  input  ‘data’.  However,  it  is  important  for  the  stakeholders 
that  as  clear  a  distinction  as  possible  should  be  made  and  managed  as  the  analysis  proceeds, 
lest  the  stakeholders  lose  confidence  in  the  coherence  and  ultimate  credibility  of  the  analysis. 

The  determination  of  validity  in  any  particular  case  will  hinge  upon  triangulation  of  some  or  all  of  these 
dimensions.  Validity  will  be  judged  greater  as  the  correlation  increases  between  the  characteristics  of  the 
methods  employed.  Triangulation  is  discussed  further  in  Sections  6.6.1  and  8.2.1. 

There  are  human  limits  to  validity  in  judgement-based  OA,  arising  from  the  impact  of  biases  and  ‘heuristics’ 
in  acts  of  judgement.  When  humans  declare  opinions,  it  can  reasonably  be  assumed  that  their  utterances  will 
not  be  fully  objective,  but  rather  that  they  will  be  vulnerable  to  bias.  Gathering  subjective  judgements  from 
experts  exposes  the  analyst  and  his  study  to  possibilities  of  two  varieties  of  bias:  motivational  and  cognitive. 
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The  first,  motivational  bias,  arises  when  an  expert’s  opinion  is  influenced  by  non-relevant  circumstances,  for 
example  a  drive  to  please  the  client  of  the  study  to  which  he  is  contributing.  This  bias  can  be  conscious  or 
unconscious;  the  expert  may  or  may  not  realise  that  he  has  altered  his  view.  The  second  is  cognitive  bias 
which  can  intrude  when  an  expert  does  not  follow  rules  or  standards  which  are  held  to  be  objectively 
derived.  An  expert  will  often  draw,  consciously  or  unconsciously,  upon  ‘heuristic’  approaches  to 
determination  of,  e.g.  a  variable’s  value.  A  good  example  is  anchoring ,  i.e.  the  reluctance  of  an  expert, 
perhaps  for  unconscious  emotional  reasons,  to  amend  an  initial  view  which  might  have  been  inadvertently 
suggested  by  a  third  party  or  even  the  analyst  himself.  Motivational  bias  may  be  very  hard  to  detect  and 
nearly  impossible  to  eliminate.  Much  depends  upon  the  attitudes  and  ethics  of  the  analysts  and  the  experts. 
Cognitive  biases,  by  contrast,  are  better  understood  and  have  been  studied  experimentally,  offering 
possibility  that  they  can  be  eliminated  or  compensated  for.  The  phenomenon  of  biases  will  be  discussed 
further  in  Chapter  7. 

It  is  helpful  to  make  a  clear  distinction  between  internal  and  external  validity  in  the  application  of 
subjective  methods;  internal  validity  refers  to  the  logical  grounding  and  coherence  of  the  proposed 
methods,  whilst  external  validity  concerns  the  congruity  of  the  methods  to  the  area  of  application.  Greater 
precision  and  control  in  the  design  and  execution  of  the  methods  will  increase  its  internal  validity.  Wider 
applicability  of  a  method  is  an  indicator  of  increased  external  validity.  In  this  sense,  validity  can  be 
thought  of  as  fitness  for  purpose. 

3.4.2  Achieving  Validity 

What  are  the  threats  to  validity? 

Internal  validity  is  typically  vulnerable  to: 

•  Poorly  defined  measurement  scales  or  categories; 

•  Statistical  weakness  resulting  from  very  poor  levels  of  participation  by  experts;  and 

•  Inadequate  recording  methods. 

External  validity  is  threatened  by: 

•  Poor  understanding  of  the  objectives  and  priorities  of  the  analysis; 

•  Misunderstanding  of  the  domain  of  expertise,  robustness  of  the  outputs  to  uncertainty,  etc.;  and 

•  The  effects  of  human  traits  such  as  the  biases  introduced  by  heuristic  reasoning  (ref.  Section  7.2). 

Some  of  the  internal  factors  will  interact  with  the  external  factors  and  this  must  be  considered  in  the  study 
design.  An  example  is  the  danger  of  mathematical  artefacts  in  the  outputs,  particularly  where  statistical 
analysis  has  been  carried  out.  Any  analyst  should  be  aware  of  these  threats  and  make  efforts  to  identify 
them,  avoid  them  or  cope  with  them. 

Overall,  a  balanced,  holistic  view  of  validity  is 
required,  and  some  compromises  will 
undoubtedly  be  needed  in  any  practical  design 
carried  out  within  limited  resources.  Complete 
validation  is  probably  not  achievable  in  all 
applications,  but,  as  with  objective  methods,  it  is  something  to  be  desired  and  aimed  for.  It  is  only  meaningful 
to  debate  validity  of  a  method  at  the  point  of  application,  i.e.  in  the  context  of  the  decision  problem  being 
addressed.  Is  the  method  to  be  used  sufficiently  useful  for  its  purpose?  Perhaps  the  biggest  threat  to 
validity  is  ontological:  knowing  at  the  design  stage  what  is  likely  to  be  important.  Only  experience  can 
tell  you.  This  threat  can  itself  be  mitigated  by  creating  an  iterative  design  similar  to  that  used  in 


It  is  only  meaningful  to  debate  validity  of  a  method  at  the 
point  of  application,  i.e.  in  the  context  of  the  decision 
problem  being  addressed. 

“All  models  are  wrong,  but  some  are  useful'’  [3]. 
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experimentation.  Having  created  a  design  based  on  a  level  one  understanding  of  the  issues  and  executed  it, 
use  the  results  to  re-formulate  priorities  and  expectations,  redesign  and  re-execute. 

3.4.3  Credibility  and  Acceptance 

Credibility  is  achieved  when  expertise  is  delivered  with  trustworthiness.  The  level  of  credibility  a  study 
method  may  have  will  depend  on  the  amount  of  expertise  available  to  apply  it,  and  the  perceptions  of  the 
client  (and  other  stakeholders)  of  that  expertise.  Indeed,  there  are  both  subjective  and  objective  components 
to  credibility.  In  particular,  the  trustworthiness  of  an  analyst  will  itself  be  subjective  and  personal  to  the 
client  system3.  In  order  to  gain  credibility  for  those  methods  which  have  been  validated  through 
consideration  of  the  above  factors,  it  is  necessary  to  also  consider  the  acceptance  of  such  models  from  the 
decision-maker  community. 

Acceptance  is  a  concept  which  is  present  throughout  the  study  process.  It  starts  at  the  point  where  the 
problematic  situation  is  raised  and  is  critical  through  to  the  end  of  the  process,  where  the  analysis  will  be 
exploited  by  the  client  system.  Communication  between  the  analyst  (i.e.  the  study  team)  and  the  client 
system  is  critical  throughout  all  aspects  of  the  study  process;  often  lack  of  communication  with 
stakeholders  can  be  missed  until  the  very  end  of  a  study  and  so  damage  its  acceptance.  Acceptance  can  be 
gained  through  transparency  and  communication  of  the  modelling  process.  This  will  help  the  analyst  gain 
the  trust  and  confidence  of  the  decision  makers  in  the  method,  and  more  importantly,  the  study  outcomes 
so  that  these  can  be  of  most  value.  Transparency  of  judgement-based  OA  is  discussed  further  in  Chapter  8. 

The  confidence  a  client  has  in  an  analyst  can  also  be  increased  through  adequate  preparation.  For  example, 
training  an  analyst  improves  their  own  confidence  which  will  then  be  more  easily  portrayed  to  the  client 
system.  Also,  ensuring  resources  and  facilities  are  available  can  ease  the  model  building  and  data  elicitation 
process. 

Credibility  and  acceptance  will  be  further  reinforced  by  independent  scrutiny. 


3.5  ETHICAL  CONSIDERATIONS 

An  analyst  needs  to  “conduct  activities  in  an  ethical  manner  that  deserves  the  confidence  of  all  parties 
involved”  [4].  Ethical  practice  requires  that  the  following  action  disciplines  must  be  observed  [5]: 

•  Act  with  skill  and  care,  and  keep  skills  up  to  date; 

•  Prevent  corrupt  practice  and  declare  conflicts  of  interest; 

•  Respect  and  acknowledge  the  work  of  others; 

•  Ensure  that  research  is  justified  and  lawful;  and 

•  Do  not  mislead  -  present  evidence  honestly. 

Very  similar  ethical  standards  are  promulgated  by  The  Military  Operational  Research  Society  (of  the  USA) 
on  its  website  [6].  Contributors  to  ‘soft’  OA-based  studies  should  acknowledge  these  ethical  guidelines  and 
those  of  any  professional  organisation  of  which  they  are  a  member. 

Note  that  these  disciplines  are  as  incumbent  on  the  stakeholders  in  general  as  on  the  analysts,  facilitators, 
etc.,  who  are  conducting  the  analysis.  It  is  possible  that  some  of  the  stakeholders  may  find  the  process  of 
analysis  and  its  results  discomfiting.  They  should  be  content  that  they  have  been  granted  ‘ procedural 


3 

The  term  ‘client  system’  refers  to  the  (quite  common)  situation  where  there  is  no  single  individual  acting  as  the  client, 
but  rather  a  group  of  individuals  or  (part)  of  an  organisation.  Figure  4-1  (Chapter  4)  depicts  the  client  system  in  red  ovals. 
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justice’’ :  they  have  been  involved,  and  can  judge  the  fairness  of  the  analysis  process,  even  though  they  may 
not  agree  with  its  outcome,  provided  that  there  was  initial  agreement  on  the  fairness  of  the  rules  followed 
during  the  collaborative  process. 

Many  ‘soft’  methods  require  information  and  data  to  be  taken  directly  from  experts  ‘in  the  field’.  The  use 
of  subjective  methods  could  well  be  viewed  as  an  experiment,  particularly  by  the  experts  from  who 
information  is  being  captured.  They  are,  in  a  sense,  the  subjects  to  which  the  methods  are  being  applied; 
their  involvement  could  be  construed  as  a  threat  to  their  (mental)  integrity.  It  is  arguable,  therefore, 
that  the  disciplines  of  ethical  experimentation  should  be  applied,  at  least  in  spirit,  to  many  of  the  subjective 
methods.  Considerations  relevant  to  this  perspective  include: 

•  Accessibility  of  the  methods  to  all  stakeholders; 

•  The  integrity  of  the  participants  themselves  in  their  respective  roles  (chairman,  member  of  workshop, 
stakeholder,  analyst,  client); 

•  The  perceived  validity  of  the  method  and  its  processes; 

•  The  consent  of  participants; 

•  The  utilisation  of  resources  including  participant/stakeholder  time; 

•  Publication  and  distribution  of  results  from  the  methods; 

•  Feedback  for  participants; 

•  Relationship  with  other  contributing  elements  and  parties  to  a  domain  being  served  by  application 
of  a  ‘soft’  method;  and 

•  Attitudes  towards  and  actions  taken  by  the  client  and  other  senior  staff  in  power  roles  to  participants 
in  aftermath  of  an  application  of  a  subjective  method. 

The  key  test  of  the  ethical  validity  of  a  method  is  whether  it  yields  fresh  knowledge;  it  would  not  be 
ethical  to  execute  a  study  method  which  did  not  generate  new  data.  It  should  be  acknowledged  that  some 
of  this  knowledge  is  frequently  of  a  sociological  kind,  reflecting  prior  ignorance  on  the  part  of  some 
stakeholders  of  the  perspectives  and  legitimate  and  well-founded  preferences  of  other  stakeholders. 
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•  Chapter  1  -  INTRODUCTION 

•  Chapter  2  -  PROBLEMATIC  SITUATIONS  AND  'SOFT  OA 

•  Chapter  3  -  ACHI EVI NG  VALI DITY,  CREDI  Bl  LITY  AND  ACCEPTANCE 


•  Chapter  4 -ROLES  AND  RESPONSIBILITIES 


•  Chapter  5  •  THE  'SOFT'  OA  STUDY  PROCESS 

•  Chapters  -  DESIGNING  THE 'SOFT' OA  STUDY  METHODOLOGY 

•  Chapter  7-  DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 

•  Chapter8  -  INTERPRETING,  USING  AND  COMMUNICATING  RESULTS  AND  OUTCOMES 


•  Chapter  9  -  THE  ESSENCE  OF  THE  CODE  OF  BEST  PRACTICE 

•AnnexA  -  LIST  OF  KEY  STATEMENTS  AND  RECOMMENDATIONS  OF  IMPORTANCE 
TO  THE  ANALYST 

•  Annex  B  -  SELECTIVE  LIST  OF  'SOFT'  METHODS 


>  Many  roles  and  responsibilities  are  undertaken  during  the  conduct  of  a  ‘soft’  OA  study.  Moreover,  one 
individual  may  adopt  different  roles  involving  different  responsibilities.  Each  individual  involved  in  the 
study  contributes  to  its  quality  and  success. 

>  The  analyst,  through  his  design  and  conduct  of  the  study  and  through  his  reporting  and  interpretation 
of  the  study’s  outcomes,  plays  a  key  role  in  contributing  to  the  study’s  validity,  credibility  and 
acceptance. 

>  The  facilitator  too,  through  his  management  of  study  process  (the  way  participants  interact),  plays  a 
key  role  in  contributing  to  the  study’s  validity,  credibility  and  acceptance. 

>  The  (primary)  client  has  a  responsibility  towards  the  analyst  in  supporting  the  execution  of  the  study 
and  should  act  as  a  partner  to  the  analyst. 

>  Good  stakeholder  management  is  the  responsibility  of  the  analyst  and  is  critical  to  study  success. 
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ORGANIZATION 


4.1  INTRODUCTION 

This  chapter  discusses  the  roles  and  responsibilities  of  the  significant  actors  who  are  associated  with 
studies  that  apply  ‘soft’  OA.  While  acknowledging  that  each  study  is  unique,  it  nevertheless  provides 
some  guidelines  to  analysts.  The  guidelines  are  offered  as  a  means  of  assistance  and  are  not  necessarily 
prescriptive  in  nature.  In  general,  this  chapter  deals  with  the  people  involved  in  a  study  that  employs  ‘soft’ 
OA  and  with  what  those  people  do,  whereas  how  those  activities  are  carried  out  is  discussed  in  the 
subsequent  chapters  of  this  CoBP. 


4.2  INDIVIDUALS  INVOLVED  IN  A  ‘SOFT’  OA  STUDY 

Individuals  associated  with  a  study  that  applies  ‘soft’  OA  may  play  the  roles  of: 

•  Analyst.  An  individual  who  conducts  a  study,  and  in  this  capacity  designs  the  stages  of  the  process 
and  suggests  ways  to  investigate  and  model  problematic  situations,  the  methodology(ies)  and 
method(s)  to  be  used  based  on  his  knowledge  and  expertise,  the  workshops  to  be  held,  and  the 
ways  to  report  and  interpret  the  study’s  outcomes. 

•  Client.  This  term  denotes  four  specific  types  of  stakeholders  (who  form  a  so-called  ‘client 
system’):  the  sponsor  who  actually  owns  the  study,  the  customer  who  pays  the  bill  for  the  study, 
the  decision  maker  who  may  make  decisions  regarding  the  problematic  situation  or  concerning  the 
recommendations  of  the  study  in  preparation  for  decisions  by  a  higher  authority  and  the  end  user 
who  is  ultimately  affected  by  the  study  outcomes  and  related  decisions.  It  should  be  noted  that: 

•  The  customer  hires  the  analyst  to  conduct  the  study  and  has  discretion  over  funds.  Sometimes, 
however,  the  party  paying  the  bill  for  the  study  has  no  stake  in  the  study  outcome. 

•  The  decision  maker  is  not  necessarily  in  command  of,  or  part  of,  the  sponsor’s  organisation. 

•  Expert  (also  known  as  a  Subject-Matter  Expert  (SME)):  An  individual  who  has  considerable 
relevant  knowledge  in  a  particular  area  without  necessarily  owning  or  otherwise  being  part  of  the 
problematic  situation.  The  SME  may  be  considered  to  be  a  stakeholder  since  he  affects  the  study 
through  the  provision  of  information.  But,  in  general,  in  his  role  of  SME,  this  individual  does  not 
necessarily  own  (part  of)  the  problematic  situation  and  need  not  otherwise  be  a  part  of  the 
problematic  situation. 

Some  SMEs  may  be  part  of  the  ‘client  system’.  End  users  may  sometimes  act  as  SMEs  as  well. 
In  some  security  domains,  the  expertise  is  possessed  by  a  limited  number  of  individuals  who  will 
therefore  find  themselves  in  more  than  one  of  the  sub-groupings  of  the  stakeholders. 

•  Facilitator.  An  individual  who  helps  (i.e.  supports,  enables  and  encourages)  individuals  or  a  group 
or  groups  of  clients,  subject  matter  experts,  other  stakeholders,  and  analysts  to  work  together 
through  the  stages  of  a  study  by  managing  procedure  (the  way  the  problem  is  tackled)  and  process 
(the  way  participants  interact)  whilst  adopting  an  impartial  attitude. 

•  Stakeholder.  An  individual  who  can  affect,  or  can  be  affected  by  the  (resolution  of  the)  problematic 
situation.  The  stakeholder  system  consists  of  individuals,  groups  and  organisations  that  are 
affected  by  the  problematic  situation.  In  the  military  context,  an  enemy  may  also  be  a  stakeholder. 
In  such  cases,  the  analyst  should  ensure  that  enemy  perspectives  and  likely  reactions  are  taken 
into  account  within  the  study  context  (e.g.  through  the  use  of  Red  Teaming).  Even  analysts  and 
facilitators  may,  to  some  extent,  be  regarded  as  stakeholders  as  they  too  affect  the  study  or  are 
affected  by  it  (e.g.  due  to  their  interest  in  the  study’s  perceived  success). 

•  Scrutineer  (reviewer,  auditor):  An  individual  who  provides  an  independent  review  of  the  study. 
Although  the  scrutineer  does  not  act  as  an  analyst,  he  may  choose  to  perform  his  own  analysis  as 
part  of  his  scrutineer  function.  The  use  of  professional  scrutineers  contributes  to  study  credibility 
and  acceptance  (see  Section  3.4.3). 
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In  designing  and  executing  the  study,  the  analyst  should  take  into  account  the  diverse  nature  of  the  client 
system.  In  particular,  the  analyst  should  appreciate  that  each  client: 

•  Has  a  will,  a  purpose  and  resources,  and  may  influence  the  study; 

•  Will  perceive  the  problematic  situation  in  a  unique  way,  based  on  his  own  perspectives, 
commitments,  objectives,  interests,  values,  desires,  and  concerns; 

•  May  have  both  a  declared  and  an  undeclared  agenda;  and 

•  May  engage  in  organisational  politics  (which  is  not  necessarily  a  bad  thing,  and  not  necessarily 
avoidable  or  to  be  avoided). 

The  analyst  deals  with  members  of  the  client  system  who  may  come  from  several  different  parts  of  an 
organisation  or  from  several  different  (possibly  external)  organisations.  Within  each  such  organisation, 
the  analyst’s  clients  may  hold  different  levels  of  responsibility  within  their  respective  organisations  and 
belong  to  various  levels  within  the  organisational  hierarchy.  The  analyst’s  clients  may  be  staff  at  a  high 
level  (e.g.  general/flag  officers)  or  at  a  working  level,  and  they  may  be  military  or  civilian. 

Each  stakeholder  will  have  different  perceptions  as  to  whether  he  and  his  organisation  will  benefit  from 
study  outcomes  and  hence  will  either  be  supportive,  neutral  or  opposed  to  the  study.  Some  stakeholders 
(even  the  clients  among  them)  may  actively  attempt  to  subvert  the  study  process  and  its  outcomes  (e.g.  try 
to  slow  down  or  stop  the  study  due  to  political  issues,  a  power  play,  a  hidden  agenda,  conflicting  goals  or 
an  unwillingness  to  admit  that  there  is  a  problem).  Each  client  will  have  some  degree  of  influence  in 
determining  whether  the  study  succeeds,  especially  as  it  relates  to  whether  the  ideas  or  recommended 
changes  emerging  from  the  study  will  be  accepted  as  being  credible  and  hence  whether  they  will  be 
implemented. 

The  situation  is  further  complicated  by  the  reality  that  a  particular  role  within  the  study  may  be  undertaken 
by  one  or  more  individuals  and  may  even  be  carried  out  by  different  individuals  at  different  times. 
Additionally,  an  individual  may  take  on  one  or  more  roles  at  any  given  time  and  his  role(s)  may  change 
over  time  as  the  needs  of  the  study  change,  as  his  perceptions  of  the  problematic  situation  change,  or  as  he 
moves  out  of  his  current  position  within  the  organisation  (e.g.  civilian  promotion,  military  posting). 

The  analyst  should  try  to  understand  the  client  system  as  it  exists  at  the  start  of  the  study  and  as  it  evolves 
over  time.  He  should  try  to  deal  appropriately  with  each  of  the  individuals,  groups  and  organisations 
involved  directly  in  the  study  or  having  a  significant  stake  in  it.  For  example,  in  designing  and  conducting 
the  study,  the  analyst  should  account  for  cases  in  which  persons  wear  several  ‘hats’  at  the  same  time 
(e.g.  a  person  who  is  the  study  sponsor  but  is  also  acting  as  a  SME  may  be  influenced  by  his  interests  as 
the  study  sponsor  when  providing  his  SME  inputs  during  a  workshop).  As  another  example,  in  designing 
and  conducting  the  study,  the  analyst  should  account  for  differences  in  culture  (e.g.  national, 
organisational,  military/civilian)  among  the  study  participants.  As  well,  he  should  be  ready  to  deal  with 
changes  in  the  membership  of  the  client  system  and  with  changes  in  the  identity  of  the  primary  client1, 
both  of  which  are  not  uncommon  during  the  conduct  of  a  study. 

Figure  4-1  graphically  depicts  individuals’  roles  and  some  of  the  possible  relationships  between  the  roles2. 
The  figure  is  not  definitive  in  nature;  as  noted  above,  changes  in  roles  may  take  place  over  time. 
This  aspect  of  the  study  should  be  managed  by  the  analyst  through  the  study  design  and  by  the  facilitator 
during  workshop  sessions  (e.g.  an  individual  may  be  asked  to  put  on  just  one  of  his  several  ‘hats’  when  he 
participates  in  a  facilitated  session). 


1  The  term  ‘primary  client’  refers  to  the  single  individual  or  (small)  group  of  individuals  who  belong  to  the  client  system  and 
who  are  acting  as  a  partner  to  the  analyst  throughout  the  conduct  of  the  study.  The  ‘primary  client’  is  the  primary  focus  of 
attention  for  the  analyst  and  is  discussed  in  Section  4.3.2. 

2 

The  size  of  the  oval  areas  within  the  figure  and  the  size  of  their  overlaps  are  purely  illustrative. 
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In  the  remainder  of  this  chapter,  discussion  is  focused  primarily  on  the  roles  and  responsibilities  of  the 
analyst  and  the  facilitator.  The  roles  of  the  other  actors  within  the  study  are  discussed  where  appropriate, 
and  primarily  from  the  viewpoint  of  the  analyst. 


4.3  THE  ANALYST3 

4.3.1  Analyst  Task 

The  analyst  [1]  should  take  responsibility  for  the  design,  execution  and  scientific  integrity  of  the  study. 

In  this  capacity,  he  should: 

•  Help  structure  and  define  the  nature  of  the  problematic  situation; 

•  Design  the  stages  of  the  process  used  within  the  study; 

•  Suggest  ways  to  investigate  and  model  the  problematic  situation  and  suggest  the  methodology(ies) 
and  method(s)  to  be  used,  the  workshops  (if  any)  to  be  held,  and  ways  to  report  and  interpret  the 
study’s  outcomes; 

•  Provide  facilitated4  modelling  (generally  through  the  use  of  workshops;  including  an  integrated 
process  of  data  gathering  and  model  creation  and  analysis  [2]); 


3  Most  of  this  section  is  relevant  to  all  OA  studies. 

4  This  is  a  process  where  the  analyst  actively  engages  the  client  (group)  to  participate  in  all  stages  of  a  ‘soft’  OA  study  and  thus 
acts  as  a  facilitating  analyst  (often  with  the  help  of  a  supporting  facilitator). 
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•  Help  develop  agreed  outputs  (e.g.  better  understanding  of  the  problematic  situation,  identification 
of  desirable  and  politically  feasible  options  for  improving  the  problematic  situation); 

•  Report  and  interpret  the  study  process  (including  participants),  results  and  outcomes; 

•  Provide  a  rational  and  logical  analysis  that  aids  in  managing  the  complexity  of  the  problematic 
situation,  aids  in  recognising  and  managing  uncertainty  and  risk,  and  adequately  meets  the  overall 
study  goals;  and 

•  Remain  devoted  to  providing  a  credible,  rigorous  analysis  and  to  ethical  principles  (as  discussed 
in  Chapter  3). 

This  section  addresses  analyst  tasks,  namely: 

•  Initial  exploration  and  planning; 

•  Stakeholder  analysis; 

•  The  choice  of  an  OA  approach; 

•  The  choice  of  study  participants;  and 

•  The  general  conduct  of  the  OA  study. 

It  also  discusses: 

•  Dealing  with  uncooperative  clients; 

•  Dealing  with  clients’  participation  requests; 

•  Building  the  analyst  team; 

•  Balancing  analyst  responsibilities;  and 

•  Analyst  skills  and  training. 

The  information  is  provided  from  the  analyst’s  perspective,  concentrating  on  the  activities  that  need  to  be 
carried  out.  Other  chapters  in  this  CoBP  (and  the  literature)  discuss  how  the  activities  should  be  carried 
out. 

4.3.2  Initial  Explorations  in  Cooperation  with  the  (Primary)  Client 

When  the  analyst  is  called  in  for  support,  one  of  the  first  matters  that  he  should  consider  is  the  nature  of 
the  task  and  the  type  of  analysis  that  would  be  appropriate.  Initially,  he  should  gather  some  basic 
information  on  the  problematic  situation  through  consultations  with  some  of  the  members  of  the  client 
system.  He  should  identify  the  primary  client  and  should  start  to  determine  whether  he  is  dealing  with  a 
puzzle,  a  problem  (proper)  or  a  mess  (defined  in  Chapter  2)  with  the  intent  of  using  this  characterisation  to 
determine  whether  the  task  should  be  addressed  through  the  use  of  ‘hard’  OA,  ‘soft’  OA  or  a  mixed 
approach  (further  discussed  in  Chapters  2  and  6).  The  analyst  should  recognise  that,  in  the  case  of  a  mess, 
the  initial  characterisation  will  naturally  be  very  limited  in  nature.  In  such  a  case,  structuring  and  defining 
the  problematic  situation  will  become  one  of  the  key  challenges  in  the  study. 

The  analyst  should  also  recognise  that,  in  addressing  a  messy  situation,  there  may  be  uncertainty  over  who 
the  clients  are  and  who  should  be  chosen  as  the  primary  client.  When  identifying  the  clients  of  the  study, 
the  analyst  should  consider  who  would  benefit  from  the  study,  who  is  seeking  solutions  to  the  same  or 
similar  problems,  and  who  will  be  affected  by  possible  study  outcomes.  The  analyst  may  expect  some 
members  of  the  client  system  to  self-identify  to  the  analyst  when  they  become  aware  of  the  study. 
However,  others  who  will  be  affected  by  the  study  will  be  unaware  of  it  and  may  not  come  forward  on 
their  own.  Some  may  even  actively  seek  to  avoid  it. 
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The  analyst  may  identify  clients  by  identifying  the  people  involved  in  the  area  under  investigation  and 
approaching  them  to  determine  whether  they  are  potential  clients  and  whether  they  know  of  others  who 
may  be.  The  analyst  should  also  ask  the  sponsor  to  help  identify  potential  clients.  He  can  then  ask  those 
who  have  been  identified  for  additional  leads.  Through  this  type  of  procedure,  the  analyst  should 
eventually  identify  the  primary  client  and/or  the  initial  client(s)  he  may  work  with.  As  more  information  is 
gathered,  new  members  of  the  client  system  may  be  identified  and  their  participation  in  the  study 
negotiated.  The  analyst  should  conduct  stakeholder  analysis  (discussed  later  in  this  section  and  in  Chapter 
6)  in  order  to  better  understand  the  client  system  and  to  develop  a  plan  for  its  management  (e.g.  whom  to 
involve  in  the  study  and  in  which  capacities,  for  example:  problem  scoping;  analysing  and  designing 
options  for  addressing  the  problematic  situation;  workshop  participant,  data  provider,  contributor  to  the 
study’s  budget). 

The  analyst  should  try  to  communicate  directly  with  potential  clients  as  opposed  to  relying  wholly  on 
assumptions  that  are  based  on  information  gleaned  from  second-hand  sources,  when  developing  his 
understanding  of  the  client  system  and  his  client  management  plan.  It  should  be  noted  that  the  analyst  may 
be  subject  to  practical  restrictions  in  his  choice  of  whom  to  involve  in  the  study,  and  how  study 
participants  may  be  dealt  with.  For  example,  the  primary  client  may  request  that  the  analyst  not  contact 
other  clients  directly,  but  rather  work  through  the  chain  of  command  when  approaching  others. 

The  analyst  should  focus  on  the  primary  client  and  should  ultimately  aim  to  satisfy  the  primary  client’s 
needs  in  terms  of  the  study.  He  should  also  consider  the  welfare  of  all  the  members  of  the  client  system 
and  should  attempt  to  meet  their  collective  needs  in  a  fair  and  rigorous  manner  to  the  extent  that  it  is 
reasonable  and  possible  to  do  so. 

Identifying  the  primary  client  may  be  a  challenging  task.  The  sponsor  may  be  the  primary  client,  but  this  is 
not  always  the  case.  For  example,  it  may  be  difficult  to  select  the  primary  client  from  within  a  client 
system  that  includes  individuals  with  very  different  perspectives  and  agendas.  As  another  example, 
the  real  primary  client  may  be  too  senior  to  have  the  time  to  participate  in  the  study,  and  an  individual  who 
can  act  as  a  surrogate  for  the  real  client  may  need  to  be  found.  Further,  the  real  primary  client  may  remain 
invisible  to  those  directly  involved  in  the  study.  Additional  complications  in  terms  of  ensuring  that  the 
study  meets  its  goals  may  arise  when  the  person  acting  as  the  primary  client  is  not  the  most  senior  person 
in  his  organisation  and  hence  needs  to  ‘sell’  the  study  process  and  its  outcomes  to  his  seniors  as  well  as  to 
convince  them  that  study  resources  are  being  well  allocated.  Despite  these  challenges,  the  analyst  should 
identify  the  primary  client  early  on  in  the  study,  possibly  through  use  of  the  approach  outlined  in  the 
preceding  section.  It  is  preferable  that  the  primary  client  be  an  individual.  If  the  choice  of  the  primary 
client  must  consist  of  a  group  of  individuals,  then  their  number  should  be  limited  to  a  few.  In  this  case, 
it  is  preferable  that  the  analyst  meet  with  the  group  members  jointly,  rather  than  individually,  whenever 
decisions  need  to  be  made.  This  helps  avoid  placing  the  analyst  in  the  position  of  trying  to  accommodate 
different,  and  possibly  conflicting,  interests  while  in  the  receipt  of  inconsistent  guidance. 

Once  he  has  identified  the  primary  client,  the  analyst  should  ensure  that  the  primary  client  that  has  been 
identified  actually  understands  and  actually  accepts  the  responsibilities  that  come  with  the  primary  client 
role. 

In  particular,  the  individual(s)  chosen  to  act  as  the  primary  client  should  accept  his  obligation  to  act  as  a 
partner  to  the  analyst.  He  should  understand  that  it  is  crucial  for  him  to  fulfil  his  duties  towards  the  analyst 
in  order  for  the  study  to  succeed.  The  analyst  may  ask  the  primary’  client  to: 

•  Adopt  ownership  of  the  study  and  its  results; 

•  Provide  his  insights  on  the  problematic  situation  (e.g.  his  current  perception  of  the  issues,  of  key 
actors,  of  the  sources  of  expertise  that  the  study  could  draw  on); 

•  Assist  the  analyst  in  developing  and  evolving  the  study  design  as  the  study  progresses; 
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•  Approve5  the  study  design  and  changes  to  it; 

•  Provide,  or  help  ensure  access  to,  SMEs  and  some  of  the  information,  data,  and  documents  needed 
during  the  course  of  the  study; 

•  Participate  in  the  study  and  encourage  other,  possibly  busy,  individuals  to  participate  in  the  study 
(e.g.  by  issuing  invitations  to  participate  in  workshop  sessions,  by  providing  an  introduction  for 
the  analyst  to  other  stakeholders,  by  building  a  high  level  of  visibility  and  support  for  the  study 
within  the  client  community);  and 

•  Provide  the  analyst  with  adequate  access  to  stakeholders  and  an  adequate  pool  of  study  participants 
(both  in  quantity  and  quality)  is  critical  for  study  success. 

The  first  meeting  with  the  primary  client  is  important  as  it  sets  the  tone  for  the  primary  client-analyst 
relationship.  The  analyst  should  build  a  relationship  with  the  primary  client  that  is  based  on  mutual  respect 
and  trust.  He  should  ensure  that  there  is  open  communication  and  that  the  primary  client  feels  free  to  ask 
questions,  challenge  assumptions,  provide  input  and  discuss  contentious  issues.  The  analyst  should 
cooperate  with  the  primary  client  and  adopt  a  helpful  attitude.  He  should  use  that  initial  meeting  to 
develop  a  shared  understanding  of  the  primary  client’s  current  expectations  with  regard  to  the  study 
(e.g.  why  the  primary  client  wants  the  study  initiated,  what  the  primary  client  intends  to  do  with  the  study 
outputs,  what  types  of  assessments  decision  makers  would  like  the  study  to  support,  and  what  types  of  end 
products  are  desired).  The  analyst  should  also  discuss  what  resources  are  available  (e.g.  personnel)  and 
should  ask  about  some  of  the  study  constraints  (e.g.  time,  organisational  culture,  organisational  politics, 
legal  issues,  physical  issues,  ethical  issues).  In  designing  the  study,  the  analyst  should  consider  the  types 
of  assessments  the  decision  maker  expects  the  study  to  help  support  and  should  also  consult  with  end  users 
to  the  extent  possible  and  practical. 

The  analyst  should  recognise  that,  in  the  early  stages  of  the  study,  the  primary  client’s  expectations  may 
not  yet  be  well  formulated.  The  initial  meeting  will  likely  yield  only  partial  answers  to  some  of  the 
analyst’s  questions.  The  analyst  should  put  the  client  at  ease  by  making  it  clear  that  he  is  not  expected  to 
have  all  the  answers.  The  analyst  should  recognise  that  the  primary  client’s  expectations  will  likely  mature 
and  crystallise  over  time  as  the  study  progresses  and  new  insights  are  developed. 

After  some  investigation,  the  analyst  should  formulate  a  study  plan  (e.g.  goals,  scope,  products,  schedule, 
and  resources),  albeit  with  the  caveat  that  the  plan  will  need  to  be  regularly  revisited,  and  perhaps 
changed,  as  the  study  evolves  (as  discussed  in  Chapters  5  and  6).  In  the  case  of  a  mess,  the  analyst  should 
ensure  that  clients  appreciate  and  accept  that  a  dynamic  approach  and  study  plan  (further  discussed  in 
Chapters  5  and  6)  may  be  needed.  In  order  to  help  ensure  that  the  study  plan  does  not  become  a  point  of 
contention  between  the  primary  client  and  the  analyst,  the  analyst  should  explain  the  necessity  for 
flexibility  and  should  negotiate  expectations  about  what  is  to  be  done,  why  it  is  appropriate  and  what 
content  and  process  milestones  will  be  met  [3]. 

The  analyst  should  consult  regularly  with  the  primary  client  and  should  design  the  study  so  that  it 
regularly  delivers  useful  outputs  to  the  client.  These  mechanisms  should  help  minimise  client  concern 
about  the  need  for  study  flexibility  and  the  lack  of  a  completely  predetermined  study  plan.  ‘Soft’  OA 
presents  many  opportunities  for  regular  output  delivery.  Its  involvement  of  the  client  community  in  the 
study  process  allows  regular  communication  of  study  progress  and  results,  unlike  studies  which  release 
their  recommendations  only  at  the  study’s  conclusion  and  the  issue  of  the  final  report. 

In  developing  the  plan  and  in  conducting  the  study,  the  analyst  should  also  ensure  that  the  primary  client 
and  other  members  of  the  client  system  appreciate  the  importance  and  value  of  both  the  tangible  (e.g.  set 
of  recommendations)  and  the  intangible  (e.g.  greater  understanding  and  changed  appreciation  of  key 


Note  that  others,  for  examples  scrutineers,  may  be  involved  in  approving  the  study  design. 
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problematic  issues  gained  through  participation  in  the  study)  outputs  that  a  study  employing  ‘soft’  OA 
may  produce.  Key  actors  who  act  as  study  participants  should  realise  that  the  knowledge  they  gain  as 
participants  helps  them  not  only  deal  better  with  the  present,  but  also  positions  them  to  deal  better  with 
new  related  issues  that  will  arise  in  the  future. 

From  the  initial  meeting  with  the  primary  client  and  other  stakeholders,  the  analyst  should  manage 
expectations.  The  expectation  levels  should  correspond  to  what  is  actually  attainable  and  agreed  upon. 

The  analyst  should  be  frank  and  open  in  his  dealings.  Fie  should  advise  the  primary  client  of  the  possible 
advantages  that  can  arise  from  a  chosen  process  and  also  of  the  possible  disadvantages  (e.g.  an  issue  that 
the  primary  client  wishes  to  keep  dormant  may  be  made  public  at  a  workshop  by  one  of  the  participants, 
the  primary  client  may  not  like  some  of  the  recommendations  that  are  generated  during  a  workshop).6 

The  analyst  should  also  provide  an  opportunity  for  the  primary  client  to  air  any  fears  or  concerns  that  he 
may  have  about  the  study.  For  example,  the  primary  client  may: 

•  Be  unsure  about  his  ability  to  provide  the  types  of  inputs  and  judgements  being  sought  from  him; 

•  Be  uncomfortable  over  the  freedoms  that  may  be  granted  to  the  analyst  within  his  home 
organisation  (and  needed  in  order  to  enable  the  analyst  to  conduct  the  study); 

•  Question  how  the  perspectives  of  those  who  have  conflicting  agendas  will  be  incorporated  into  the 
study; 

•  Be  afraid  of  losing  control  or  power; 

•  Be  concerned  that  hidden  agendas  will  be  uncovered;  or 

•  Be  concerned  over  budget  reductions  that  may  result  from  the  study. 

The  analyst  should  address  issues  proactively  and  directly  in  order  to  ensure  that  client  concerns  do  not 
adversely  affect  the  study. 

As  stated  earlier,  the  analyst  should  recognise  that  defining  the  problematic  situation  itself  will  be  a 
challenge.  The  primary  client  may  just  have  a  vague  sense  that  the  problematic  situation  needs  improvement 
but  may  not  be  aware  of  his  objectives  and/or  priorities,  and  he  may  be  hoping  that  the  analyst  will  provide 
help  in  developing  them.  Or,  the  primary  client  may  confuse  a  symptom  that  he  is  observing  as  being  his  real 
problem  and  may  need  the  analyst  to  help  uncover  what  the  real  issues  are.  Alternately,  it  may  not  be  clear 
who  the  key  actors  are  and  who  the  analyst  should  involve  in  the  problem  structuring  and  definition.  As  well, 
the  client  system  itself  may  be  comprised  of  powerful  individuals  with  conflicting  interests  and  perspectives, 
presenting  difficulties  in  client  management  and  in  meeting  client  needs.  To  address  these  issues,  the  analyst 
should  consider  the  use  of  Problem  Structuring  Methods  (PSMs)  and  stakeholder  analysis  (discussed  in  the 
following  section). 

4.3.3  Stakeholder  Analysis 

To  apply  ‘soft’  OA  successfully,  the  analyst  should  develop  a  good  understanding  of  the  social  system 
(i.e.  roles,  norms  and  values  of  the  key  actors)  and  the  political  system  (i.e.  how  power  is  expressed, 
obtained,  protected,  preserved,  passed  on  and  relinquished;  who  holds  power;  explicit  and  hidden  agendas) 
in  which  he  is  operating  [4].  Finding  out  about  the  social  aspects  of  the  situation  may  be  difficult  as 
participants  may  not  want  to  admit  what  their  real  norms  and  values  are  or  they  may  not  want  the  culture 
to  change  and  so  will  not  be  willing  to  help  with  the  analysis.  Direct  questions  may  result  in  participants 
providing  politically  correct  answers  or  official  myths  rather  than  a  true  depiction  of  what  is  really 


6  A  ‘hard’  OA  study  may  have  similar  issues. 
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happening.  Carrying  out  these  activities  constitutes  stakeholder  analysis  and  decisions  on  what  to  do  with 
the  stakeholders  constitute  the  management  plan. 

The  analyst  should  conduct  the  stakeholder  analysis  as  discussed  in  Chapter  6  as  part  of  the  problem 
framing  process.  He  should  regularly  update  it  during  the  study  [3],  preferably  on  a  proactive  basis. 
The  need  to  regularly  revisit  and  revise  the  stakeholder  analysis,  and  hence  also  the  stakeholder 
management  plan  (which  is  a  product  of  the  stakeholder  analysis),  stems  from  the  stakeholder  community 
not  being  static.  As  the  study  progresses  through  its  lifecycle  the  members  within  the  stakeholder 
community  may  change  and/or  the  perceptions  of  individuals  may  change. 

The  analyst  addresses  the  purpose,  resources,  intent,  power  and  will  of  the  stakeholders  in  order  to  make 
decisions  on  how  to  address  the  stakeholders  within  the  study  context.  The  analyst  should  use  the 
stakeholder  analysis  to: 

*  Identify  stakeholders; 

•  Gain  an  understanding  of  each  stakeholder’s  motivations  and  how  each  stakeholder  fits  into  the 
overall  study  context; 

*  Develop  a  plan  as  to  how  stakeholders  should  be  engaged  and  managed  during  the  study  (e.g.  how 
much  attention  to  devote  to  each  stakeholder,  which  stakeholders  to  invite  to  workshops,  which 
stakeholders  to  include  in  formal  communications  regarding  study  outputs,  how  to  manage 
pluralistic  competing  stakeholder  demands);  and 

•  Identify  potential  risks  to  the  study  (e.g.  uncooperative  stakeholders  and  their  possible  adverse 
effects  on  the  study)  and  ways  to  address  those  risks. 

Examples  of  management  strategies  that  the  analyst  may  choose  to  employ  include,  but  are  not  limited  to 
keeping  a  stakeholder  informed,  having  a  stakeholder  actively  participate  in  the  study,  trying  to  influence  a 
stakeholder,  and  monitoring  a  stakeholder.  When  deciding  which  stakeholders  to  include  as  participants  in 
workshops  and/or  other  study  aspects,  the  analyst  should  consider  ethical,  practical,  legal,  security  and 
policy  issues.  For  example,  the  analyst  may  decide  to  exclude  adversarial  stakeholders  from  participating 
in  the  study.  However,  the  analyst  should  consider  including  some  of  the  stakeholders  with  divergent 
interests  as  this  may  be  useful  in  ensuring  that  the  problematic  situation  is  viewed  holistically  from 
multiple  perspectives,  that  key  issues  are  not  missed  and  that  study  outputs  are  credible.  The  analyst  may 
need  to  keep  the  stakeholder  analysis  private  as  the  primary  client  may  be  uncomfortable  with  an  explicit 
record.  As  well,  the  stakeholder  plan  may  not  be  universally  distributed  as  some  stakeholders  may  not 
accept  that  they  should  be  ‘managed’.  In  conducting  the  stakeholder  analysis,  the  analyst  should  identify 
stakeholders  at  a  level  of  granularity  (e.g.  individual,  organisation)  at  which  they  can  be  influenced  and 
managed.  The  analyst  should  exercise  good  stakeholder  management  throughout  the  life  of  the  study. 

4.3.4  Choice  of  the  OA  Approach 

The  analyst  should  recognise  that  the  choice  of  the  OA  approach  may  also  present  challenges.  The  analyst 
should  accept  that  the  final  responsibility  for  that  choice  rests  with  him. 

The  analyst  should  make  the  choice  of  the  OA  approach  based  on  the  characteristics  of  the  problematic 
situation  and  the  study  outputs  that  are  to  be  produced.  He  should  consider  the  pragmatic  aspects  of  the 
situation  such  as  whether  the  client  culture  supports  the  use  of  the  proposed  methods,  the  cognitive 
feasibility  of  the  proposed  methods  (e.g.  can  study  participants  reasonably  be  expected  to  meet  the 
cognitive  task  demands  associated  with  the  implementation  of  the  proposed  methods).  He  should  also  take 
into  account  analyst  skills,  analyst  preferences,  analyst  experience,  and  analyst  competencies  [5]. 
The  analyst  should  make  the  choice  in  consultation  with  the  primary  client  and  possibly  other  members  of 
the  client  system,  using  the  constructs  described  in  Chapter  6.  When  dealing  with  a  mess,  the  analyst  may 
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sometimes  organise  a  group  discussion  with  select  stakeholders  regarding  the  choice  of  methodology  and 
methods,  and  the  pros  and  cons  of  the  alternatives. 

If  a  client  suggests  that  a  particular  method  be  used,  then  the  analyst  should  exercise  caution.  He  should 
determine  why  this  suggestion  is  being  made  and  assess  its  merits.  Is  the  method  one  which  the  person 
making  the  recommendation  has  used  successfully  in  the  past  and  trusts,  or  is  there  some  reason  for  the 
suggestion  other  than  the  appropriateness  of  the  method? 

The  analyst  should  make  choices  regarding  the  analytical  approach  at  the  study’s  outset,  but  should  not 
consider  them  as  fixed.  Interventions  change  client  perceptions  and  hence  change  the  nature  of  the 
problem  being  addressed  and  the  nature  of  the  analysis  required.  The  application  of  ‘soft’  OA  tackles 
problems  in  a  dynamic  way  in  which  the  development  of  understanding  and  learning  is  episodic.  As  well, 
as  methods  are  applied,  new  issues  may  arise  that  were  not  anticipated  earlier. 

The  analyst  should  continually  reflect  on  the  different  dimensions  of  the  problem  space  (e.g.  physical,  social, 
individual),  on  the  intellectual  resources  available,  on  the  process  (e.g.  legitimacy,  potential  sources  of  error 
and  bias)  and  on  the  outcomes  in  order  to  ensure  that  an  appropriate  combination  of  methodologies,  methods 
and  processes  are  being  applied  to  address  the  situation  at  hand.  He  should  evolve  his  choices  (e.g.  research 
design,  data  collection,  data  representation  and  process)  based  on  critical  reflection  on  what  is  happening 
during  the  course  of  the  study  and  what  remains  to  be  accomplished  in  order  to  produce  the  types  of  outputs 
agreed  to  with  the  primary  client  and  needed  by  the  client  system  [6]. 

4.3.5  Choice  of  Study  Participants 

The  analyst  should  recognise  that  decisions  regarding  study  participants  shape  the  study  and  the  acceptance, 
credibility  and  validity  of  its  outputs.  The  analyst  should  make  decisions  regarding  the  choice  of  study 
participants  based  on  the  topic  being  addressed  and  on  participant  knowledge,  expertise,  experience, 
availability  and  willingness  to  contribute.  He  should  try  to  ensure  that  all  key  viewpoints  are  adequately 
represented  within  the  study  process.  Overlooking  and/or  excluding  key  clients  from  the  study  risks  study 
failure.  Clients  who  are  not  properly  engaged  within  the  study  context  may  not  feel  ownership  (e.g.  ‘buy- 
in’)  towards  study  outcomes  and  may  act  against  the  implementation  of  any  resultant  recommendations. 
The  analyst  should  try  to  ensure  that  client  involvement  is  such  that  the  mechanisms  of  interaction 
encourage  adequate  ownership  of  the  study  at  the  required  levels  of  the  client  system  hierarchy.  As  such, 
the  analyst  should  carefully  manage  the  type  and  extent  of  participation  by  each  individual.  He  should 
consider  regularly  communicating  study  outputs  to  select  individuals  who  choose  not  to  participate  more 
fully  in  the  study  process  but  whose  relationship  with  the  study  is  considered  important. 

The  analyst  should  carefully  consider: 

•  The  number  of  SME  participants  that  are  required  during  each  stage  of  the  study; 

•  Which  individual  SMEs  are  best  suited  to  act  as  study  participants; 

•  The  means  to  be  used  to  elicit  information  from  the  SMEs;  and 

•  How  the  SME-generated  information  is  to  be  validated  and  used  (further  discussed  in  Chapter  7). 
For  example,  the  analyst  may  ask  a  SME  to  participate  in  one  or  more  of  the  following  activities: 
gathering  information,  structuring  issues,  designing  options,  evaluating  options  and  external 
review  of  the  study  as  a  whole  or  select  aspects  of  it. 

More  specifically,  the  analyst  may  ask  a  SME  to  participate  in  one  or  more  of  the  following  activities: 

•  Provide  his  personal  judgement; 

•  Act  as  a  representative  of  his  organisational  group  and  hence  provide  judgements  in  terms  of  the 
group’s  official  stance; 
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•  Attempt  to  simulate  the  viewpoints  of  a  select  adversarial  stakeholder  (e.g.  Red  Teaming); 

•  Provide  judgements  based  on  the  current  state  of  affairs; 

•  Make  predictions  concerning  the  future; 

•  Act  as  a  free  thinker  without  considering  policy  or  other  constraints;  and 

•  Provide  judgements  within  the  context  of  being  bound  by  current  policy  or  other  select  constraints. 

For  example,  the  analyst  may  ask  a  SME  to  contribute  assumptions  regarding  future  doctrine,  performance 
data,  force  mixes,  organisational  structures,  or  force  employment  scenarios.  A  SME  may  be  unwilling, 
or  even  unable,  to  perform  some  or  all  of  the  activities  that  the  analyst  asks  him  to  engage  in.  Furthermore, 
a  SME  may  sometimes  believe  that  some  of  the  activities  that  he  is  being  asked  to  perform  are  in  conflict. 
He  may  for  these,  or  other  reasons  (e.g.  lack  of  time  to  devote  to  the  study),  refuse  to  perform  some  or  all 
of  the  roles  that  he  is  being  asked  to  carry  out. 

In  choosing  SME  participants,  the  analyst  should  recognise  that  while  a  SME  does  have  detailed 
knowledge,  it  is  not  likely  to  be  complete  and  could  be  biased,  whether  intentional  or  not  (as  discussed  in 
Chapters  3  and  7). 

Circumstances  may  dictate  whether  one  or  more  SMEs  should  be  employed  in  a  study  that  uses  ‘soft’  OA. 
In  some  cases,  SMEs  may  be  very  busy  and  it  may  be  a  challenge  to  find  SMEs  to  participate  in  a  study 
(for  example,  due  to  the  paucity  of  SME  in  some  security  domains).  As  such,  analysts  may  sometimes  be 
disappointed  to  find  that  the  persons  assigned  to  act  as  SMEs  do  not  have  the  required  knowledge, 
experience  or  expertise  to  carry  out  their  designated  task.  They  may  also  find  it  challenging  if  there  is  a  need 
for  continuity  in  the  involvement  of  SMEs  within  the  study  context  but  different  SMEs  show  up  at  each 
meeting.  Sometimes,  the  analyst  may  find  it  necessary  to  bring  in  additional  SMEs  from  the  end  user, 
customer  or  even  analyst  communities.  The  analyst  should  try  to  employ  a  mix  of  SMEs  who  together 
adequately  represent  the  key  viewpoints  from  which  the  topic  under  discussion  may  be  considered  and  who 
are  accepted  by  their  peers  as  being  experts  in  the  field  under  consideration.  Employing  such  a  mix  of  SMEs 
may  help  alleviate  some  of  the  concerns  regarding  SME  bias  and  impartiality  by  way  of  triangulation. 

In  general,  the  analyst’s  design  choices  should  encourage  client  ownership  of  the  study.  The  analyst 
should  attempt  to  build  a  good  relationship  with  study  participants  and  the  members  of  the  client  system  in 
general  that  is  based  on  the  clients’  confidence  in  the  analyst’s  integrity,  honesty,  skills,  judgement  and 
fairness.  He  should  recognise  that  his  study  design  choices,  such  as  the  choice  of  study  participants  and  the 
manner  in  which  judgements  are  elicited  (discussed  in  Chapter  7),  directly  affect  the  quality  of  the  study’s 
outputs  [7]. 

4.3.6  Conducting  the  Study 

The  quality  of  a  study  relies,  to  a  large  extent,  on  the  analyst’s  qualifications,  experience,  perspective, 
training,  track  record,  status,  and  presentation  of  self  [8].  A  skilled,  experienced  and  mature  analyst  is 
more  likely  to  make  good  choices  of  methodologies  and  methods  for  use  in  the  study. 

In  designing  and  executing  a  study,  the  analyst  should  carry  out  the  whole  of  the  study  jointly  with  the 
primary  client.  The  analyst  should  meet  regularly  with  the  primary  client.  He  should  ensure  that  the 
primary  client  is  kept  satisfied  in  terms  of  the  study’s  progress,  as  well  as  the  relevance  and  credibility  of 
its  outputs. 

In  conducting  the  study,  the  analyst  should  rely  on  the  primary  client’s  problem  domain  knowledge  and 
understanding  of  the  context  and  on  his  own  professional  OA  expertise  and  skills.  He  should  use  a  process 
that  is  iterative  in  nature  with  repeated  reformulation  of  the  problematic  situation  and  study  approach  as  new 
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insights  are  gained.  The  analyst  should  typically 
employ  the  phased  approach  discussed  in 
Chapter  5  (i.e.  appreciation,  analysis,  assessment 
and  action).  He  should  orchestrate  the  elicitation 
and  use  of  information  and  data  from  select 
stakeholders  (discussed  in  Chapter  7)  for  the 
framing  of  the  problematic  situation  and  for 
the  development  of  options  for  addressing 
the  problematic  situation,  as  well  as  for  their 
communication  and  interpretation  (discussed  in  Chapter  8). 

The  analyst  should  recognise  that  there  may  be  no  one  right  solution  to  the  complex  issues  being 
addressed  by  the  study.  He  should  recognise  that  the  use  of  subjectivity  in  generating  recommendations  is 
unavoidable  [2].  Furthermore,  in  the  case  of  a  mess,  the  best  one  may  hope  for  may  be  to  better  manage 
(rather  than  to  solve)  the  mess  that  is  being  addressed. 

The  analyst  should  recognise  that  he  plays  a  critical  role  in  the  study’s  credibility,  validity  and  acceptance. 
He  should  accept  his  responsibility  to  provide  a  rational  and  logical  analysis  that  aids  in  managing  the 
complexity  of  the  problematic  situation  and  aids  in  recognising  and  managing  uncertainty  and  risk. 
He  should  conduct  the  study  in  a  way  that  accords  to  the  best  practices  in  the  field  (e.g.  standards  of 
practice  set  out  by  relevant  scientific  societies,  professional  bodies,  his  own  organisation)  and  to  ethical 
principles  (further  discussed  in  Chapters  3,  5  and  6).  Many  organisations  have  ethics  boards  to  ensure  that 
the  research  being  undertaken  conforms  to  standards  and  have  documents  to  guide  their  researchers. 
Common  quality  issues  that  the  analyst  should  address  (discussed  in  Chapter  3)  concern,  but  are  not 
necessarily  limited  to  ([9],  [10]): 

•  Choice  of  participants; 

•  Handling  of  bias; 

•  Clear  account  of  criteria  used  to  select  cases  and  data  collection  and  analysis; 

•  Clear  theoretical  assumptions; 

•  Systematic  data  collection  and  record  keeping; 

•  Documentation  of  the  rationale  behind  the  research  design  and  data  analysis; 

•  Clear  discussion  of  models  employed; 

•  Reference  to  accepted  procedures  for  analysis; 

•  Detailed  list  of  evaluative  criteria  (methods  appropriate  to  question  asked,  connection  to  existing 
body  of  knowledge); 

•  Use  of  reliable  data; 

•  Findings  supported  by  data; 

•  Clear  links  between  data  interpretations  and  conclusions;  and 

•  Clear  distinction  between  data  and  its  inteipretation. 

In  conducting  the  study,  the  analyst  should  try  to  ensure  procedural  justice  (i.e.  agreement  on  the  fairness  of 
the  rules  followed  during  collaboration)  and  procedural  rationality  (i.e.  the  rationality  of  the  means  followed 
to  achieve  set  goals)  during  the  conduct  of  the  study  [3].  Generally  the  analyst  should  try  to  keep  all  the 
members  of  the  client  system  satisfied,  within  reason.  However  in  practice  it  may  prove  impossible,  given 
differences  in  interests  and  values  and  conflicting  personal  agendas,  to  develop  outputs  from  the  study  that 
satisfy  everyone  or  it  may  even  be  undesirable  to  do  so  in  the  context  of  the  adversarial  positions  of  some  of 


A  seminar  war  game  (SWG)  on  the  ‘Army  of  Tomorrow” 
examined  military  operations  of  the  future  by  incorporating 
both  military  and  civilian  (e.g.  police,  diplomats,  aid 
agencies,  scientists,  and  others)  judgements  and 
perspectives.  It  provided  a  means  to  ‘meld’  these  diverse 
judgements  together  in  order  to  produce  insights  into  future 
operational  challenges  and  opportunities. 

The  ‘soft’  OA  approach  provided  a  means  to  engage 
stakeholders  who  came  from  different  cultures  and  had 
very  different  points  of  view. 
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the  stakeholders.  The  analyst  should,  as  a 
minimum,  try  to  keep  the  primary  client  satisfied 
in  terms  of  the  execution  of  the  study. 

In  conducting  the  study,  the  analyst  should  also 
aim  for  transparency.  For  example,  he  should 
ensure  that  the  study  process  is  well 
documented  and  auditable.  The  analyst  should 
attempt,  to  the  best  of  his  ability,  to  act  in  an 
unbiased  manner  when  addressing  the  differing 
perspectives  and  interests  related  to  the 
problematic  situation.  Readers  are  referred  to 
Chapters  3  and  8  for  a  discussion  of  analyst 
bias,  study  participant  bias  and  study  validity, 
credibility  and  acceptance.  It  is  preferable  for 
the  analyst  to  be  independent  and  to  have  no 
personal  interest  in  the  ultimate  outcome  of  the 
study,  but  this  is  not  always  possible  in  practice.  In  the  end,  the  study’s  credibility  needs  to  be  established 
among  all  stakeholders,  with  decision  makers  having  enough  confidence  in  the  study’s  outputs  to  use  them 
to  inform  the  decisions  that  they  make.  The  integrity  of  the  analyst  and  his  actions  during  the  study 
directly  affect  the  quality  of  the  study. 

As  part  of  the  conduct  of  the  study,  the  analyst  is  also  responsible  for  reporting  and  interpreting  study 
results  and  outcomes.  This  activity  is  discussed  in  Chapter  8. 


During  the  planning  and  conduct  of  a  seminar  war  game 
(SWG)  concerning  the  ‘Army  of  Tomorrow’,  the  clients 
were  intimately  involved  in  all  phases  related  to  the 
operational  employment  of  the  SWG.  The  analysts  acted 
as  advisors.  This  approach  resulted  in  the  clients 
developing  a  feeling  of  ownership  towards  the  SWG  which, 
in  turn,  helped  avoid  many  potential  hazards.  For  example, 
this  approach  helped  ensure  that  participants  were  well 
prepared  to  assume  their  assigned  roles  on  the  day  of  the 
SWG  and  made  it  more  likely  that  the  outputs  of  the  SWG 
were  viewed  as  being  credible.  The  analysts  were 
responsible  for  the  rigour  of  the  process.  Analyst  guidance 
was  normally  well  accepted  by  the  clients  due  to  the  need 
to  ensure  the  credibility  and  acceptance  of  the  SWG 
outputs  in  the  eyes  of  outsiders. 

The  application  of  ‘soft’  OA  was  based  on  the  mutual 
respect  and  trust  between  the  clients  and  the  analysts,  and 
on  the  analysts  striving  to  have  the  clients  take  as  much 
ownership  of  the  SWG  process  as  possible. 


4.3.7  Dealing  with  an  Uncooperative  Client 

On  occasion,  the  analyst  may  need  to  deal  with  an  uncooperative  client.  Several  approaches  are  possible. 
In  some  cases,  the  analyst  may  decide  to  exclude  the  client  from  the  study  but  to  attempt  to  take  his 
perspectives  and  possible  actions  into  consideration,  and  to  mitigate  the  damage  that  such  an  individual 
could  do  to  the  study.  This  may  be  done,  for  example,  by  having  another  knowledgeable  client  provide 
input  on  what  the  likely  perspectives  and  reactions  of  the  uncooperative  client  are.  Alternately,  the  analyst 
may  try  to  turn  an  uncooperative  client  into  a  neutral  or  supportive  client  through  frank  discussion, 
lobbying  or  negotiation. 

The  analyst  should,  however,  be  aware  that  some  uncooperative  clients  may  not  be  readily  apparent. 
Analysts  should  be  careful  of  clients  who  attempt  to  ‘game’  study  activities  in  an  attempt  to  force  outputs 
to  conform  to  their  perspective  of  the  problem  (e.g.  manipulation  of  weights  when  voting  on  options  in  a 
multiple  criteria  decision  analysis  context).  When  manipulations  in  voting  are  suspected,  the  analyst 
should  document  the  study  process  to  provide  an  audit  trail  and  should  consider  conducting  an  analysis  of 
the  voting  of  sub-groups. 

Analysts  should  also  be  careful  of  stakeholders  who  act  as  workshop  participants  in  order  to  gather 
evidence  that  they  intend  to  use  to  torpedo  the  study  if  their  favourite  issues  show  poorly.  The  analyst 
should  also  be  careful  of  situations  in  which  the  primary  client  wants  advocacy,  especially  since  some 
clients  with  such  intent  may  not  make  it  explicit;  the  analyst  needs  to  be  firm  with  such  clients.  Negative 
stakeholders  should  be  consciously  addressed,  even  if  the  ultimate  decision  is  to  ignore  them.  Readers  may 
wish  to  consult  [11]  for  a  further  discussion  about  working  with  (difficult)  groups  (see  also  the  discussion 
in  Section  7.3  and  its  Table  7-1). 
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4.3.8  Dealing  with  Clients’  Participation  Requests 

Occasionally,  the  analyst  may  encounter  higher  level  clients  (e.g.  general/flag  officers)  who  wish  to 
participate  on  a  part-time  basis  in  a  workshop.  This  allows  these  clients  to  gain,  and  contribute  to,  a  much 
richer  appreciation  of  the  problematic  situation  without  needing  to  make  significant  time  commitments. 
It  also  provides  them  with  an  opportunity  to  confirm  that  the  study  is  on  time  and  on  budget  and  to  develop 
confidence  in  the  study  outputs.  Furthermore,  such  limited  participation  may  lead  to  the  identification  of 
others  who  may  gainfully  be  brought  into  the  study  (e.g.  a  fonner  commander,  a  representative  from  an 
Allied  Service,  an  academic  who  has  also  been  studying  the  problem). 

Despite  its  advantages,  the  analyst  should  be  aware  that  such  participation  may  also  negatively  affect 
workshop  dynamics  (e.g.  due  to  differences  in  rank  among  participants;  further  discussed  in  Chapter  7) 
if  not  handled  properly.  Normally,  the  analyst  should  encourage  such  participation  but  should  manage  it  so 
that  it  benefits  both  the  individual  and  the  study.  The  facilitation  style  may  need  to  be  amended  to  mitigate 
some  of  the  potential  negative  effects  of  such  participation. 

Occasionally  the  analyst  may  encounter  a  stakeholder  who  requests  to  attend  a  session  purely  as  an 
observer.  The  analyst  may  discourage  this  practice  as  it  can  negatively  affect  the  energy  of  the  workshop 
and  its  ultimate  success.  Alternatively,  the  analyst  may  allow  this  practice  but  may  carefully  manage  it  in 
order  to  minimise  its  potentially  adverse  effects.  For  example,  the  analyst  may  manage  such  requests  by 
scheduling  all  observers  to  attend  only  on  a  select  day  of  a  multi-day  session. 

4.3.9  Analysis  Teams 

The  analyst  may  work  alone  or  as  a  member  of  a  team7,  on  a  full  or  part-time  basis.  He  may  provide  both 
analytical  and  facilitation  services,  or  may  concentrate  on  the  analysis  (e.g.  model  creation  and  analysis) 
while  working  closely  with  one  or  more  facilitators  and/or  other  analysts.  If  the  study  calls  for  the  use  of 
specialised  skills  that  the  analyst  does  not  have  (e.g.  specialised  ‘soft’  OA  methods  expert),  he  should  use 
the  expertise  of  other  analysts  or  step  aside  to  be  replaced  by  another  analyst  who  has  the  required 
specialised  skills.  The  size  and  nature  of  the  task,  and  whether  a  mixed  OA  approach  is  required,  should  be 
used  to  determine  whether  the  use  of  an  analytical  team,  headed  by  a  team  leader,  is  appropriate. 
If  external  facilitators  are  to  be  used,  then  the  analyst  should  consider  bringing  them  into  the  study  early 
on  as  this  provides  facilitators  a  better  opportunity  to  provide  direct  and  valuable  input  into  the  study 
design  and  execution. 

4.3.10  Balancing  Responsibilities 

The  analyst  may  find  it  challenging  to  transition  between  and  balance  his  responsibilities  as  the  analyst  for 
the  study,  a  researcher  in  the  OA  domain,  a  mentor  of  junior  OA  practitioners  and,  in  some  instances, 
as  facilitator  for  the  study’s  workshops.  He  should  address  this  challenge  by  being  a  reflective  practitioner 
and  should  act  with  pragmatism.  He  should  continually  review  his  own  range  of  OA  knowledge  and  skills 
and  further  his  own  methodological  competence.  He  should  dedicate  himself  to  professional  development, 
keeping  abreast  of  new  developments  in  his  field  and  ameliorating  his  analytical  ‘toolbox’  from  the 
learning  opportunities  that  arise.  He  should  actively  mentor  and  nurture  the  development  of  OA  skills 
among  more  junior  practitioners  with  the  understanding  that  it  is  his  professional  responsibility  to  do  so. 
His  research  should  inform  his  practice,  and  his  practice  should  inform  his  research.  In  this  way,  work 
undertaken  in  a  study  not  only  benefits  the  study’s  clients,  but  also  supports  the  advancement  of  the  OA 
profession  including  its  theoretical  underpinnings.  Naturally,  the  analyst  should  always  respect  the  client’s 
privacy  needs  when  determining  how  insights  gleaned  from  real-world  applications  of  ‘soft’  OA  are 
published  in  scientific  journals. 


7  The  analyst  team  works  in  conjunction  with  the  larger  study  team  whose  composition  varies  by  study  (and  over  time) 
according  to  the  study’s  needs. 
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4.3.11  Analyst’s  Skills  and  Training 

In  common  with  many  professions,  there  are  no  prescribed  criteria  stating  the  set  of  competencies  that  an 
analyst  employing  ‘soft’  OA  should  possess.  However,  general  experience  has  provided  insights  on  the 
skills  that  are  required.  The  analyst  employing  ‘soft’  OA  typically  needs  to  have  the  skills  of  both  analyst 
(modeller)  and  facilitator  (although  sometimes  a  study  using  ‘soft’  OA  employs  individuals  purely  in  an 
analytical,  or  purely  in  a  facilitative,  role).  Individuals  who  are  good  at  ‘applied  scientific  common  sense’ 
and  in  handling  group  dynamics  will  likely  be  good  at  ‘soft’  OA. 

In  terms  of  training,  knowledge  and  skills,  it  is  generally  desirable  that  the  analyst  employing  ‘soft’  OA 

([1],  [12]): 

•  Has  training  in  some  scientific  discipline  (though  not  necessarily  in  the  exact  sciences)  and  has  a 
well-rounded  scientific  background  (as  ‘soft’  OA  is  based  on  the  use  of  a  scientific  (and  possibly 
a  multi-disciplinary)  approach); 

•  Has  knowledge  of  some  ‘soft’  methods  and  active  experience  with  their  application; 

•  Has  the  ability  to  use  effectively  a  wide  range  of  techniques  to  understand  and  support  the  social 
and  cognitive  needs  of  participants  in  a  group  situation  in  which  a  problem  is  being  explored  and 
solutions  are  being  developed  (e.g.  active  listening,  chart-writing,  managing  group  dynamics  and 
power  shifts,  reaching  closure); 

•  Has  the  ability  to  use  software  appropriate  to  the  field  of  application; 

•  Can  deal  tactfully  with  the  concerns  of  others;  and 

•  Has  strong  oral  and  written  communications  skills. 

In  terms  of  interest,  personality  and  conduct,  it  is  generally  desirable  that  the  analyst  employing  ‘soft’  OA 

[1],  [12]]): 

•  Is  interested  in  practical  solutions; 

•  Can  take  a  broad  holistic  view  of  the  issue  under  study; 

•  Possesses  a  high  degree  of  initiative,  energy,  and  maturity; 

•  Is  a  ‘self-starter’  who  can  work  well  individually  and  in  a  team; 

•  Has  a  high  degree  of  general  intelligence  and  enthusiasm  for  the  work; 

•  Enjoys  drawing  knowledge  from  other  disciplines  (as  OA  is  interdisciplinary  and  may  draw  solutions 
from  engineering,  psychology,  and  other  social  and  political  sciences); 

•  Has  the  right  personality  so  as  to  be  able  to  gain  client  confidence  and  effectively  communicate 
study  results  to  senior  decision  makers; 

•  Is  devoted  to  scientific  rigour  and  ethical  principles;  and 

•  Has  adopted  and  shows  a  fundamentally  helpful  attitude. 

It  may  also  be  useful  if  an  analyst  employing  ‘soft’  OA  has  additional  specialist  training  and  knowledge 
appropriate  to  the  field  of  application  (e.g.  advanced  analytical  methods  such  as  mathematical  modelling. 
However,  this  type  of  expertise  may  also  be  made  available  to  a  study  through  the  involvement  of  an 
additional  analyst  who  specialises  in  the  required  skill  set. 

Although  much  can  be  learned  from  books,  ‘soft’  OA  skills  are  best  developed  through  practice  and  much 
is  learned  ‘on  the  job’.  Apprenticeship  under  a  seasoned  practitioner  can  be  most  beneficial  to  those  new 
to  the  subject.  Some  well-founded  ‘soft’  OA  methodology  schools  exist,  usually  based  at  universities, 
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where  training  courses  are  offered.  Many  are  found  in  the  United  Kingdom  where  aspects  of  ‘soft’  OA 
may  form  part  of  an  OA  university  degree.  An  analyst,  who  starts  out  by  conducting  studies  that  employ 
‘hard’  OA  and  who  then  acquires  ‘soft’  OA  skills  may  be  well  suited  to  conducting  studies  that  use  mixed 
methods  and  to  leading  a  multi-disciplinary  analysis  team. 

Different  ‘soft’  methods  have  different  approaches  and  styles  of  modelling.  Practitioners  tend  to  prefer 
certain  types  of  OA  approaches  over  others  based  on  their  background  disciplines,  personalities,  skills  and 
personal  preferences.  In  practice,  there  are  limitations  in  every  OA  practitioner’s  analytical  ‘toolbox’  and 
there  is  a  trade-off  between  breadth  and  depth  of  knowledge.  Some  practitioners  are  more  able  to  be  fluent 
in  multiple  genres  of  analysis  than  others.  Particularly  challenging  is  the  ability  to  be  adept  at  employing  a 
mixed  approach  that  combines  the  use  of  ‘soft’  and  ‘hard’  OA. 


4.4  THE  FACILITATOR 

4.4.1  Facilitator  Task 

Within  ‘soft’  OA,  it  is  accepted  that  problematic  situations  are  socially  constructed  [2]  though  this  may  not 
be  the  perspective  of  the  problem  owners.  That  is,  key  actors  negotiate  about  ‘facts’,  social  relationships 
and  viable  and  acceptable  means  of  addressing  the  problematic  situation  ([13],  [14]).  A  study  employing 
‘soft’  OA  typically  uses  a  highly  participative  mode  of  engagement  and  typically,  but  not  always,  involves 
the  use  of  facilitated  sessions.  Such  sessions  may  use  formal  models  to  handle  communication  about 
content  and  to  work  with  that  content  [2].  Workshops  within  the  context  of  a  study  employing  ‘soft’  OA 
may  be  used  to  help  stakeholders  reach  agreement  on  what  the  problem  is  and  on  what  the  viable 
(satisfactory  vice  optimal)  options  for  addressing  it  are.  The  facilitator,  depending  on  the  choice  of 
methods,  may  work  at  the  individual  (e.g.  interviews)  or  group  (e.g.  workshops)  level. 

The  facilitator  should  serve  as  a  catalyst  who 
helps  the  study’s  (workshop)  participants 
manage  their  tasks  and  achieve  their  objectives. 

In  particular,  over  the  conduct  of  a  study  the 
facilitator  should  manage  process  and  procedure 
in  a  way  that  better  enables  participants  to  use 
their  own  competencies  to: 

•  Develop  a  better  understanding  of  the 
problematic  situation; 

•  Generate  options; 

•  Assess  the  options;  and 

•  Formulate  a  set  of  recommendations 
or  a  strategy  for  the  way  ahead. 

A  study  may  involve  several  workshops,  with 
each  specific  ‘soft’  OA  workshop  having  a  limited  aim. 

The  facilitator  should  also  help  participants  develop  commitment  and  ownership  of  the  group’s  outputs. 
He  should  help  them  avoid  reaching  premature  closure  (e.g.  ‘groupthink’)  that  may  lead  to  unstable 
solutions.  The  facilitator  should  attend  to  content  (the  subject  matter  under  discussion)  only  to  the  extent 
that  process  management  needs  to  be  informed  by  content.  He  should  also  manage  the  practicalities  that 
are  related  to  organising  and  running  a  group  session. 

The  facilitator  should  attend  to  the  personality,  power  and  political  issues  that  emerge  during  a  facilitated 
session.  For  example,  he  should  manage  participation  problems  and  protect  individuals,  such  as  in  cases 


A  study  conducted  for  the  Navy  on  ‘New  Operational 
Concepts  for  Maritime  Mine  Counter  Measures’  involved 
addressing  a  multitude  of  aspects  and  drawing  on  many 
different  areas  of  expertise.  The  study  was  conducted 
through  the  use  of  a  series  of  project  stages.  Facilitation 
played  a  key  role  in  helping  study  participants  stay  focused 
while  addressing  the  complexity  of  the  problem.  Facilitation 
also  helped  ensure  that  participants  were  aware  of  the 
particular  perspective  by  which  the  current  project  stage 
was  to  be  looked  upon.  The  facilitator  explained  current 
discussion  issue(s),  repeated  crucial  current  assumptions, 
clarified  the  current  activity’s  position  in  the  chain  of 
activities  being  undertaken,  ensured  that  conclusions  were 
not  adopted  without  analysing  their  rationale,  managed 
expectations  and  managed  the  changing  views  regarding 
ways  of  structuring  and  using  the  base  material.  The 
facilitator  worked  closely  with  the  analyst  to  progress  the 
study. 
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when  the  group  is  attempting  to  suppress  unpopular  or  minority  views.  Issues  may  arise,  for  example, 
if  workshop  participants  are  military  officers  of  different  ranks  or  if  workshop  participants  belong  to 
different  levels  of  the  organisation’s  hierarchy  (e.g.  one  individual  has  direct  power  over  the  other). 
The  facilitator  may  offer  protection  in  the  form  of  an  intervention  during  a  workshop  setting  or  in  the  form 
of  the  workshop  design  (e.g.  running  separate  workshops  for  higher  and  lower  ranking  individuals,  using 
computers  when  gathering  information  and  when  carrying  out  voting  over  options  in  order  to  assure  some 
degree  of  anonymity). 

The  facilitator  should  ask  clarifying  questions  and  address  miscommunication  within  the  group  [15]. 
He  should  monitor  both  verbal  and  non-verbal  cues  and  try  to  be  sensitive  to  unexpressed  feelings  in  order 
to  determine  where  significant  tension  exists.  He  should  help  the  group  manage  anxiety,  frustration  and 
tension  in  a  constructive  way. 

It  should  be  noted  that  the  study  outcome  is  affected  not  only  by  the  amount  of  conflict  present  but  also  by 
the  type  of  conflict  present  and  by  how  the  conflict  is  handled.  Conflict  may  be  the  reason  for  starting  a 
study  and  may  be  present  throughout  its  conduct.  The  type  of  conflict  present  may  be  affective  conflict 
(i.e.  socio-emotional  conflict,  personal  conflict)  that  is  rooted  in  interpersonal  relations  within  the  group 
and/or  it  may  be  substantive  or  cognitive  conflict  that  is  related  to  the  group  task  and  generally  involves 
differences  of  opinion  or  viewpoint  [15].  The  former  may  be  more  difficult  to  handle  (e.g.  individuals 
refuse  to  talk  about  issues  or  refuse  to  work  together).  The  facilitator  and  analyst  need  to  recognise  that 
conflict  is  not  always  dysfunctional  if  it  is  handled  properly.  Further,  they  should  recognise  that 
sometimes  positive  change  can  only  occur  through  conflict.  In  general,  the  facilitator  should  aim  to  do  a 
good  job  in  managing  the  emotional  life  of  the  group  [16]. 

The  remainder  of  this  section  addresses  facilitator  style  and  the  facilitator’s  contribution  to  study  credibility, 
validity  and  acceptance.  It  also  discusses  the  building  of  the  facilitator  team  and  the  requirements  that  the 
facilitator  needs  to  meet  in  terms  of  content-related  knowledge  (i.e.  knowledge  about  the  subject  matter 
under  discussion).  The  discussion  is  provided  from  the  analyst  and  facilitator  perspectives,  concentrating  on 
the  activities  that  need  to  be  carried  out  and  referring  to  other  chapters  in  this  CoBP  for  a  discussion  on  how 
the  activities  may  be  carried  out. 

4.4.2  Facilitator  Style 

There  is  no  one  correct  way  to  facilitate.  There  are  many  facilitative  styles.  The  facilitator  should  adopt  a 
situational  approach:  one  which  accommodates  the  task,  the  time  available,  the  facilitator’s  own 
personality,  the  nature  of  the  group,  and  the  dynamics  of  the  particular  combination  of  participants. 
The  facilitator  should  be  flexible  given  the  unpredictability  of  how  a  group  session  will  evolve.  He  should 
act  as  a  role  model  for  group  behaviour  as  his  behaviour  has  a  strong  impact  on  how  participants  act 
individually  and  on  the  group  dynamics.  For  a  discussion  on  common  workshop  designs  and  facilitation 
techniques  that  may  be  used  to  address  challenges  in  group  dynamics,  readers  are  referred  to  the  vast 
literature  on  facilitation,  including  for  example  some  of  the  references  consulted  in  the  writing  of  this 
section,  namely  ([10],  [15],  [16],  [17],  [18],  [19]). 

4.4.3  Facilitator’s  Contribution  to  Study  Credibility,  Validity  and  Acceptance 

The  facilitator’s  job  is  a  challenging  one.  The  facilitator  may  need  to  deal  with  heated  debates  between 
powerful  personalities  who  have  strongly  divergent  viewpoints  and  interests  regarding  the  issues  under 
discussion  and  who  are  blatantly  resisting  the  facilitator’s  attempts  to  maintain  control  of  the  session. 
A  badly  facilitated  session  in  which  the  facilitator  loses  credibility  and  the  trust  of  the  participants  may  not 
only  fail  to  produce  the  required  workshop  output  but  may  also  destroy  the  primary  client’s  confidence  in 
the  study.  The  facilitator’s  conduct  has  a  direct  impact  on  the  success  of  a  workshop  and  the  study  as  a 
whole. 
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Whatever  facilitative  style  is  used,  the  facilitator  should  try  to  ensure  that  none  of  the  participants  have 
concerns  over  his  impartiality.  Hence,  to  the  extent  possible,  the  facilitator  should: 

•  Perform  his  role  in  an  unbiased  fashion; 

•  Remain  emotionally  uninvolved; 

•  Keep  an  open  mind; 

•  Not  favour  any  participant; 

•  Encourage  equal  participation;  and 

•  Refrain  from  contributing  content,  substantive  knowledge  or  an  opinion  on  the  subject  under 
discussion;  this  includes  refraining  from  making  verbal  and  non-verbal  evaluations  regarding 
content. 

It  is  preferable  that  the  facilitator  have  no  stake  in  the  ultimate  outcome  of  the  process  (other  than 
maintaining  his  professional  credibility  throughout  that  process). 

The  facilitator  should  regularly  engage  in  critical  reflection  on  the  process  and  outcomes  in  order  to  adapt 
his  approach  to  best  suit  the  dynamically  changing  nature  and  needs  of  the  group  and  better  enable 
participants  to  manage  their  tasks  and  achieve  their  objectives. 

4.4.4  Facilitator  Team 

Although  the  analyst  (modeller)  may  typically  act  as  facilitator  during  a  study  that  employs  ‘soft’  OA, 
this  need  not  be  the  case.  Sometimes,  it  is  useful  to  split  the  analysis  and  facilitation  tasks.  This  may  be 
due  to  workload  issues  or  a  desire  to  address  the  semblance  of  impartiality  in  cases  where  the  analyst 
belongs  to  one  of  the  clients8.  In  such  cases,  it  may  be  desirable  to  bring  in  one  or  more  outside 
consultants  to  act  as  facilitators. 

The  facilitator  may  work  alone  or  as  part  of  a  team  of  facilitators.  He  may  be  external  or  internal  to  the 
organisation  to  which  the  analytical  team  belongs.  He  may  be  supported  during  a  workshop  session  by 
specially  designated  recorders.  When  the  analysis  and  facilitation  tasks  are  separated,  the  facilitator  should 
cooperate  closely  with  the  analyst  in  the  context  of  a  facilitated  modelling  approach.  It  is  not  wise  for  the 
facilitator  to  be  part  of  the  client  system  due  to  possible  concerns  about  the  semblance  of  impartiality  that 
may  arise. 

4.4.5  Facilitator’s  Content-Related  Knowledge 

The  facilitator  should  ensure  that  he  has  enough  knowledge  about  the  subject  matter  being  discussed  and  the 
acronyms  to  be  used  during  a  workshop  so  that  he  knows  enough  to  keep  track  of  the  flow  of  the  discussions, 
to  ask  the  right  questions  and  to  effectively  guide  group  processes  and  procedures.  It  generally  helps,  but  is 
not  absolutely  necessary,  for  the  facilitator  to  have  significant  content-related  knowledge  (provided  that  the 
knowledge  does  not  compromise  his  impartiality  as  a  facilitator).  The  facilitator  should  ensure  that  he  has  an 
adequate  knowledge  of  modelling  in  order  to  be  able  to  properly  support  the  particular  facilitated  modelling 
approach  that  is  being  used  during  the  workshop.  The  level  of  training  needed,  and  knowledge  of  the  subject 
matter  being  discussed,  that  is  required  for  proper  facilitation  in  support  of  the  study  should  be  considered  by 
the  analyst  when  he  chooses  the  facilitator(s)  to  be  employed  by  the  study.  Sometimes,  there  is  a  need  for 
both  a  technical  facilitator  (i.e.  knowledgeable  about  the  subject  matter  (content)  under  discussion;  or  able  to 
operate  supporting  software)  and  a  process  facilitator  (i.e.  knowledgeable  in  the  application  of  a  specialised 
OA  method)  in  a  group  setting.  The  role  of  technical  facilitator  is  often  discharged  by  an  expert  who  is 


Sometimes,  if  an  analyst  belonging  to  a  client  system  is  also  chosen  to  act  as  facilitator,  concerns  over  facilitator  impartiality 
may  be  addressed  by  ensuring  that  workshop  participants  are  chosen  such  that  they  adequately  represent  the  diverse 
stakeholder  views  and  interests. 
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imported  to  act  as  a  technical  guru  and  point  of  reference  if  technical  uncertainties  arise.  This  also  allows  the 
process  facilitator  to  detach  from  the  client  in  those  cases  where  there  are  limited  specialist  personnel 
resources  for  the  study. 

4.5  SUMMARY  OF  THE  ANALYST’S  AND  FACILITATOR’S 
RESPONSIBILITIES 

The  following  sections  provide  a  summary  of  the  analyst  and  facilitator  responsibilities  that  are  discussed 
in  this  chapter. 

4.5.1  Analyst  Responsibilities 

>  Take  responsibility  for  the  design,  conduct  and  scientific  integrity  of  a  study  involving  the  use  of 
‘soft’  OA. 

>  Provide  a  rational  and  logical  analysis  that  aids  in  managing  the  complexity  of  the  problematic  situation, 
aids  in  recognising  and  managing  uncertainty  and  risk,  and  adequately  meets  the  overall  study  goals. 

>  Continually  reflect  on  the  different  dimensions  of  the  problem  space  (e.g.  physical,  social,  individual), 
on  the  intellectual  resources  available,  on  the  process  (e.g.  legitimacy,  potential  sources  of  error  and 
bias)  and  on  the  outcomes  in  order  to  ensure  that  an  appropriate  combination  of  methodologies, 
methods  and  processes  are  being  applied  to  address  the  situation  at  hand. 

>  Identify  one  individual,  or  at  most  a  small  number  of  individuals,  within  the  client  system  as  the 
analyst’s  primary  client. 

>  Encourage  client  ownership  of  the  study. 

>  Build  a  relationship  with  the  primary  client  that  is  based  on  respect,  trust  and  open  communication. 
Adopt  a  helpful  attitude. 

>  Regularly  update  the  stakeholder  analysis  and  exercise  good  stakeholder  management  throughout  the 
conduct  of  the  study. 

>-  Ensure  that  all  key  viewpoints  are  adequately  represented  within  the  study  process. 

>  Ensure  procedural  justice  (i.e.  agreement  on  the  fairness  of  the  rules  followed  during  collaboration) 
and  procedural  rationality  (i.e.  the  rationality  of  the  means  followed  to  achieve  set  goals)  during  the 
study’s  conduct. 

>  Ensure  transparency  of  the  process  and  the  analysis  and  ensure  that  the  study  process  is  well  documented 
and  auditable. 

>  Carefully  consider  which  Subject-Matter  Experts  (SMEs)  to  involve  as  study  participants,  the  means 
employed  to  elicit  information  from  the  SMEs,  and  how  the  SME -generated  information  is  to  be  validated 
and  used. 

>  Continually  review  own  range  of  OA  knowledge  and  skills  and  further  own  methodological  competence. 

>  Use  the  expertise  of  other  analysts  if  the  study  calls  for  the  use  of  specialised,  unfamiliar  skills. 

>  Actively  nurture  the  development  of  OA  skills  among  more  junior  practitioners. 
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4.5.2  Facilitator  Responsibilities 

>  Help  (i.e.  support,  enable,  encourage)  study  participants  to  work  together  through  the  stages  of  a  study 
by  managing  procedure  (the  way  the  problem  is  tackled)  and  by  managing  process  (the  way  participants 
interact). 

>  Perform  this  role  in  an  unbiased  fashion,  remain  emotionally  uninvolved  and  refrain  from  contributing 
content  or  substantive  knowledge  on  the  subject  under  discussion.  (It  is  recognised  that  every 
individual  has  biases.  The  guidance  in  this  chapter  refers  to  ultimate  goals  that  an  individual  may 
aspire  to  achieve  to  the  extent  that  the  best  of  his  abilities  allows  him  to  do  so.) 

>  Act  impartially  and,  preferably,  have  no  stake  in  the  ultimate  outcome  of  the  process. 

>  Prior  to  conducting  a  workshop,  ensure  having  adequate  knowledge  about  the  subject  matter  being 
discussed,  the  acronyms  that  are  likely  to  be  used  and  the  results  that  the  analyst  wishes  to  generate 
through  the  workshop. 

>  Use  the  expertise  of  other  facilitators,  if  necessary.  For  example,  a  study  may  employ  a  team  of  two 
facilitators,  one  with  extensive  content-related  knowledge  (i.e.  knowledge  of  the  subject  under  discussion) 
and  another  who  has  specialised  OA  process  skills. 

>  Address  the  personality,  power  and  political  issues  that  emerge  during  a  workshop.  Protect  individual 
participants  being  singled  out,  for  example  in  cases  when  the  group  is  attempting  to  suppress  unpopular  or 
minority  views. 
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•  Chapter  1  ■  INTRODUCTION 

•  Chapter  2-  PROBLEMATIC  SITUATIONS  AND  'SOFT  OA 

•  Chapter  3  ■  ACHIEVING  VALIDITY,  CREDIBILITY  AND  ACCEPTANCE 


•  Chapters  ROLES  AND  RESPONSIBILITIES 


•  Chapter  5  -  THE  'SOFT'  OA  STUDY  PROCESS 

•  Chapter  6  ■  DESI GNING  THE  'SOFT  OA  STUDY  METHODOLOGY 

•  Chapter  7  ■  DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 

•  Chapter  8  ■  INTERPRETING,  USING  AND  COMMUNICATING  RESULTS  AND  OUTCOMES 


•  Chapter  9  ■  THE  ESSENCE  OF  THE  CODE  OF  BEST  PRACTI CE 

•  Annex  A  •  LIST  OF  KEY  STATEMENTS  AND  RECOMMENDATIONS  OF  IMPORTANCE 

TO  THE  ANALYST 

•  Annex  B  ■  SELECTIVE  LIST  OF  'SOFT'  METHODS 


>  Consider  designing  the  study  as  a  process  with  the  following  phases: 

•  Appreciation  -  ‘What  is  happening’? 

•  Analysis  -  ‘Why  and  how  is  it  happening’? 

•  Assessment  -  ‘How  can  it  happen  in  a  better  way’? 

•  Action  -  ‘What  needs  to  be  done’? 

>  Ensure  that  the  study  team  and  the  (group  of)  client(s)  reach  an  agreement  about  the  following 
concepts  at  the  end  of  each  of  the  4A-phases: 

•  Study  plan  (<-  Appreciation  phase), 

•  Problem  understanding  (<-  Analysis  phase), 

•  Set  of  options  (<- Assessment  phase), 

•  Action  plan  (<-  Action  phase). 

>  Be  aware  that  the  people  involved  in  the  process  belong  to  one  of  the  following  groups,  though  they 
should  cooperate  with  each  other: 

•  Study  Team-,  group  of  people  who  ‘do  the  work’  (e.g.  the  facilitator, 

the  analysts,  the  SMEs,  and  subsets  of  other  stakeholder  types  including  clients); 

•  Client(s):  group  of  decision  makers  (possibly  including  sponsors  and  customers). 

>  Be  aware  of  the  iterative  (cyclic)  and  dynamic  nature  of  the  process  and  its  phases. 

>  Recognise  the  divergent  or  convergent  nature  of  subsequent  phases  and  accommodate  for  it. 

>  Ensure  that  the  plan  for  application  of  ‘soft’  methods  is  understood  by  participants  and  clients  as  part 
of  their  programme  of  progressive  development.  The  same  goes  for  the  outputs  and  their  utility. 

>  Provide  a  framework  which  allows  decision  makers  to  carry  through  their  decision-making  processes 
in  a  structured,  auditable  way. 

>  Create  a  roadmap  depicting  achievements  and  interrelations,  perhaps  in  a  facilitated  workshop. 


RTO-TR-SAS-087 


5-1 


THE  ‘SOFT’  OA  STUDY  PROCESS 


ORGANIZATION 


5.1  INTRODUCTION 

This  chapter  recommends  in  the  application  of  ‘soft’  OA  that  a  study  be  conducted  as  a  dynamic  iteration 
cycle.  As  ‘real-world  problems’  are  inevitably  multi-dimensional  and  therefore  require  a  number  of  issues  to 
be  dealt  with  and  a  number  of  people  to  involve,  a  solution  approach  to  these  types  of  problems  is  usually  not 
a  single  event  but  a  process  organised  as  a  number  of  subsequent  phases  [1],  This  chapter  will  address  each 
of  these  phases.  It  will  also  address  the  iterative  nature  of  the  process  and  its  stages  of  divergence  and 
convergence. 


5.2  HOW  PEOPLE  INFLUENCE  THE  ‘SOFT’  OA  STUDY  PROCESS 

Processes  that  support  human  decision  making  within  messy  problem  spaces  and  involve  the  participation 
of  several  decision  makers,  very  often  concentrate  on  satisficing  rather  than  pure  optimisation.  Study 
procedures  applied  in  these  situations  should  support,  in  addition  to  the  appreciation  (ref.  Section  5.3.1) 
and  analysis  (ref.  Section  5.3.2)  of  the  problematic  situation  [2]: 

•  The  discovery  of  alternative  options; 

•  The  development  of  acceptable  solutions  or  ways  forward  to  improve  the  problematic  situation; 

•  The  systematic  gathering  and  analysis  of  information;  and 

•  The  use  of  bounded  rationality1  that  recognises  people’s  cognitive  limitations. 

This  implicates  two  central  elements  in  problem  resolution  processes:  negotiation  and  social  relationships 
[2].  Personal  interrelations  as  well  as  individually  diverging  agendas  and  objectives  of  the  participants 
influence  the  problem  resolving  process  and  necessitate  negotiation  efforts  in  order  to  reach  a  satisfactory 
solution. 

In  general,  the  processes  of  using  ‘hard’  and  ‘soft’  OA  approaches  to  problems  are  not  necessarily 
fundamentally  different  from  each  other.  The  challenge  of  achieving  an  effective  complementary  approach 
that  combines  ‘soft’  and  ‘hard’  OA  methods  lies  within  the  blending  of  people,  process,  and  analysis  skills 
that  suits  the  given  problem  structure  and  supports  the  achievement  of  a  common  understanding  with  and 
of  the  clients  involved  [2]. 

The  choice  of  (‘soft’/’hard’)  methods  cannot  be  fixed  at  the  beginning  of  a  study  but  evolves  from  a  critical 
reflection  of  the  developments  during  the  study  [5]. 

Critical  success  factors  for  the  study  are  the  development  among  the  analysts,  clients,  and  further  stakeholders 
involved  into  the  problem  resolving  process  of  [6]: 

•  A  shared  understanding; 

•  A  sense  of  common  purpose;  and 

•  A  commitment  to  the  way  forward. 

This  intense  involvement  of  a  variety  of  people  in  the  decision-making  process  coupled  with  the  necessity 
for  strong  communication,  coordination,  and  facilitation  amongst  them,  may  be  regarded  as  one  of  the  major 
differentiating  characteristics  between  the  application  of ‘soft’  and  ‘hard’  OA  methodologies. 


1  The  term  ‘bounded  rationality’  was  introduced  by  Herbert  A.  Simon  [3],  It  describes  the  inability  of  human  beings  to  optimise 
rationally  their  decisions  due  to  the  complexity,  dynamics,  and  ambiguity  of  decision  environments.  For  a  certain  (personal) 
constraint  system,  the  decision  maker  will  rather  satisfice  than  optimise  within  his  system  due  to  a  lack  of  important 
information  concerning  the  definition  of  the  problem  or  of  relevant  criteria  [4];  he  will  stop  his  optimisation  process  when  a 
‘satisficing’  solution  (that  may  not  necessarily  be  the  optimum)  has  been  reached. 
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Defining  an  iterative  decision  cycle  that  synchronises  the  points  of  view  of  both  the  study  team  and  the 
client  team  by  making  use  of  experienced  facilitating  analysts  is  an  effective  method  to  create  useful 
approaches  to  solving  problematic  situations.  Iteration  in  the  process  creates  the  flexibility  to  adapt  the 
problem  perception  and  to  reconsider  the  methodological  approach.  Furthermore,  it  encourages  the  people 
involved  in  the  decision-making  process  to  rethink  their  approaches  and  perceptions. 

Dealing  with  problematic  situations  that  require  the  extensive  use  of  human  judgement  in  the  decision¬ 
making  process  needs  a  process  that  is  dynamic  and  open  to  changes  caused  by  new  insights  or  changing 
minds.  An  iterative  process  that  supports  the  factors  described  above  is  elaborated  in  the  following  sections. 

5.3  THE  PHASES  OF  A  ‘SOFT’  OA  STUDY  PROCESS 

The  analyst  is  responsible  for  developing  the  study  design  (e.g.  process,  stages,  sessions,  interviews)  and 
for  adapting  (or  fine-tuning)  it  as  the  study  proceeds2.  This  includes  making  decisions  regarding  whether 
to  apply  a  combination  of  ‘soft’  and  ‘hard’  OA,  and  in  what  manner3.  The  clients’  perceptions  and  reasons 
for  commissioning  the  study  influence  choices  regarding  the  study  design.  For  example,  some  clients  may 
express  a  preference  that  quantitative  results  should  be  part  of  the  study  outputs. 

At  the  outset  of  ‘soft’  OA  studies,  analysts  do  not  ‘set  in  stone’  the  predetermined  methods,  techniques 
and  procedures  that  will  be  applied  in  a  pre-established  order  to  address,  or  produce  a  solution  to, 
the  problematic  issues  under  investigation.  The  application  of  ‘soft’  OA  tackles  problems  in  a  dynamic 
way  in  which  the  development  of  understanding  and  learning  is  non-linear  and  which  requires  a  flexible 
and  adaptive  approach  to  analysis.  As  such,  the  analyst  continually  and  critically  reflects  on  the  different 
dimensions  of  the  problem  space,  on  the  intellectual  resources  available,  on  the  process  (e.g.  legitimacy, 
potential  sources  of  error  and  bias)  and  on  the  outcomes  in  order  to  ensure  that  an  appropriate  combination 
of  methodologies,  techniques  and  processes  are  being  applied  to  address  the  situation  at  hand  and  to  find 
ways  in  which  the  process  could  be  made  even  more  efficient  and  effective.  This  is  part  of  the  iterative 
behaviour  that  the  CoBP  seeks  to  encourage. 

In  order  to  preserve  the  possibility  of  adapting  the  problem  structure  identified  and  with  it  the  chosen 
methodological  approach  and  model,  an  iterative  decision  support  process  is  advised.  This  process  is  divided 
into  phases  that  follow  each  other  sequentially  but  not  in  a  ‘one-way’  manner;  if  indicated,  earlier  phases 
can  be  repeated  and  will  then  influence  the  follow-up  phases. 

A  distinction  should  be  made  between  the  following  four  main  phases  which  helps  design  the  study4 
(based  on  [1]  and  [7]).  These  ‘4  A-phases’  (as  referred  to  in  this  CoBP)  are  composed  of: 

•  Appreciation  -  Activities  to  sense  the  problematic  situation,  identify  all  stakeholders  and  their 
viewpoints  and  assumptions,  and  other  circumstantial  factors  (‘what  is  happening’). 

•  Analysis  -  Activities  to  actually  analyse  and  structure  the  problematic  situation,  further  develop 
and  clarify  all  assumptions,  aspects,  perspectives  and  influencing  factors  and  their  (causal) 
relationships.  It  includes  provision  of  information  (data),  possibly  developing  and  relying  on  one 
or  several  models  as  representation  constructs  to  create  deeper  understanding  based  on  the  model 
outputs  (‘why  and  how  is  it  happening’). 

•  Assessment  -  Activities  to  develop,  investigate  and  compare  ways  of  change  and  improvement 
and  estimate/predict  their  consequences,  and  recommend  one  or  more  ways  ahead  (‘what  is  the 
consequence  of  what  is  happening  and  how  can  it  be  changed  to  generate  better  consequences’). 

2 

A  detailed  analysis  of  the  roles  and  responsibilities  of  the  people  involved  in  the  ‘soft’  OA  process  is  the  subject  of  Chapter  4 
of  the  CoBP. 

3 

This  issue  is  a  central  aspect  of  Chapter  6  of  this  CoBP. 

4  The  four  phases  could  help  design  any  OA  study. 
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•  Action  -  Activities  to  arrive  at  a  final  conclusion  (or  even  decision)  and  a  recommendation  of  how 
implementation  should  be  conducted  (‘what  needs  to  be  done’). 

The  four  phases  listed  above  are  similar  to  the  (military)  ‘Observation,  ‘Orientation’,  ‘Decision’  and  ‘Action’ 
loop  (OODA  loop* 1 2 3;  Figure  5-1).  The  analyst  should  for  each  of  these  main  phases  determine  its  aim  and 
what  is  required  in  terms  of  study  activities.  This  should  be  based  on  the  study’s  ultimate  aim  and  in 
agreement  with  the  client(s)  including  the  key  stakeholders. 


Figure  5-2  shows  the  iterative  process  model  of  ‘soft’  OA  which  -  after  initiating  the  study  -  consists  of 
the  four  main  phases  described  above  and  tasks  which  are  assigned  to  either  the  ‘study  team’  or  the 
‘client(s)’.  The  process  cycles  between  the  two  teams  and  relevant  information  and  decisions  are 
developed  during  facilitated  meetings  (see  Chapter  7). 


5  Or  ‘Boyd  cycle’  after  its  inventor  USAF  COL  J.R.  Boyd;  see  also  [8].  In  the  field  of  ‘situational  awareness’  a  similar  planning 
cycle  exists  using  three  levels  [9]: 

1)  Level  1  -  perception  of  the  elements  in  the  environment  (i.e.  gather  the  elements  to  build  up  a  model); 

2)  Level  2  -  comprehension  of  the  current  situation  (i.e.  validate  the  model  as  reflecting  the  common  picture  of  the  present); 
and 

3)  Level  3  -  projection  of  future  status  (use  the  model  to  forecast  consequences  of  possible  interventions). 

As  a  last  example,  the  ‘soft’  methodology  ‘Strategic  Choice’  distinguishes  between:  shape,  design,  compare  and  choose. 
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Figure  5-2:  The  Iterative  ‘Soft’  OA  Study  Cycle. 


The  methodology  of  problem  structuring  (‘Problem  Structuring  Methods’,  PSM),  encompassing  a  large 
number  of  specific  methods,  is  the  predominant  methodology  used  to  understand  a  problematic  situation, 
identify  the  concepts  that  are  relevant  and  structure  the  relationships  between  them.  They  can  therefore  be 
used  in  the  phases  of  appreciation  and  analysis.  Although  the  term  suggests  that  it  stops  at  this  point,  many 
PSM  allow  progress  toward  the  design  of  options  and  actions. 

Decision  making  regarding  problematic  situations  often  involves  a  variety  of  stakeholders6  (ref.  Chapter 
4);  the  iterative  process  depicted  in  Figure  5-2  considers  this  fact  by  assigning  tasks  to  the  study  team  and 
the  client(s). 

The  study  team  (people  who  actually  work  on  the  problematic  situation  including  participating  clients  and 
other  stakeholders)  and  the  client  team  (clients,  particularly  sponsors  and  other  people  from  the  client 
system,  who  have  to  make  decisions  about  progress  and  deliverables)  have  to  work  collaboratively 
towards  a  common  goal. 

In  each  of  the  four  A-phases  of  the  process,  both  groups  have  to  perform  certain  tasks  that  will  at  specific 
synchronisation  points  (milestones)  converge  into  an  agreement  between  the  two;  this  agreement  builds 
the  basis  for  the  further  process  work.  If  it  is  not  possible  to  reach  a  common  understanding  between  the 
two  groups  it  may  be  necessary  to  return  to  an  earlier  phase  and  refine  or  reconsider  the  work  done  in  that 


6  Analysts  and  facilitators  are  sometimes  not  considered  to  be  part  of  the  group  of  stakeholders  within  a  decision-making 
process  (ref.  Chapter  4)  but  they  are  an  integral  part  of  the  study  team. 
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phase.  Through  this  mechanism,  the  common  decision-making  process  stays  dynamic  and  iterative. 
This,  on  the  one  hand,  may  lead  to  a  longer  process  turnaround  time  and  increased  complexity  caused  by 
the  communication  and  coordination  efforts.  On  the  other  hand,  this  approach  supports  the  development  of 
an  exact  and  detailed  problem  understanding  between  the  study  team  and  the  client(s)  which  is  a  necessary 
condition  in  order  to  generate  an  appropriate  action  plan  as  a  result  of  the  decision-making  process. 

The  nature  of  the  four  A-phases  is  twofold  (Figure  5-2).  First,  they  are  primary  drivers  to  progress  the 
study  and  provide  waypoints  for  agreed  deliverables.  In  addition,  they  should  be  used  as  activities  within 
these  phases  to  refine  each  of  the  phases  in  an  iterative  manner.  Thus  the  Analysis  phase  may  include  the 
techniques  applicable  to  the  Appreciation,  Assessment  and  Action  phases  in  producing  the  agreed 
deliverable  of  the  Analysis  phase.  This  is  illustrated  in  Figure  5-2  by  the  small-sized  A-phases  at  the 
secondary  level  (Figure  5-6). The  remainder  of  this  section  will  discuss  the  main  phases  and  the  tasks  that 
are  carried  out  in  each  main  phase.  Section  5.4  will  discuss  the  iteration  and  parallel  processing  aspects, 
also  shown  in  Figure  5-2,  more  in-depth. 

The  first  process  step  that  starts  off  the  (global)  ‘soft’  OA  process  is  the  initiation-step.  This  process  step 
is  conducted  just  once,  in  contrast  to  the  following  process-steps  that  may  be  iteratively  repeated.  After  the 
initiation  of  the  process  of  ‘soft’  OA  the  following  issues  have  become  clear: 

•  A  specific  problem  or  a  problematic  situation  exists  and  needs  to  be  resolved; 

•  A  (group  of)  ciient(s)  exists  who  need(s)  decision  support; 

•  A  study  team  has  been  contracted  by  the  client(s);  and 

•  A  study  will  be  conducted  to  provide  decision  support. 

Having  settled  these  initial  issues,  the  ‘study  team’  and  the  ‘client(s)’  enter  the  first  phase  of  the  ‘soft’  OA 
study  process. 

5.3.1  The  Appreciation  Phase 

The  first  phase  of  the  iterative  process  of  ‘soft’  OA  is  the  Appreciation  phase.  The  central  task  within  this 
phase  is  to  orient  on  the  problematic  situation,  thereby  gaining  an  understanding  of  the  people  involved  as 
well  as  the  problem’s  environment. 

For  the  study  team,  the  Appreciation  phase  is  divided  into  the  three  following  tasks: 

•  Orientation  on  the  problematic  situation; 

•  Stakeholder  analysis;  and 

•  Study  design. 

The  central  task  for  the  client(s)  lies  in  providing  the  ‘Specification  of  the  study  focus’. 

This  phase  requires  an  intense  interchange  between  the  study  team  and  the  clients  in  order  to  create  a  clear 
picture  of  the  problem  environment  as  well  as  of  the  stakeholders  involved. 

As  a  result  of  this  phase,  a  commonly  agreed  study  plan  should  be  developed  which  comprises  the  answers 
to  the  ‘why?’,  ‘what?’,  ‘who?’,  and  ‘when?’  of  the  study  (adapted  from  [10]): 

•  Why  is  the  problem  addressed  by  the  study? 

•  What  tasks,  work  packages  will  be  completed  by  the  study? 

•  Who  will  be  involved  and  what  will  be  their  responsibilities  within  the  study? 
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•  When  will  which  tasks  be  performed  and  in  which  order?  What  is  the  study  timeline? 

The  study  plan  is  a  formal,  approved  document 
used  to  guide  both  study  execution  and  study 
control.  The  primary  goal  of  the  study  plan  is  to 
document  planning  assumptions  and  decisions, 
to  facilitate  communication  among  stakeholder, 
and  to  document  approved  scope,  cost,  and 
schedule  baselines  (adapted  from  [11]). 

Although  relevant  to  all  studies,  this  is  of 
special  relevance  to  ‘soft’  OA  studies 
considering  the  importance  of  communication 
and  documentation  of  changed  assumptions,  etc. 


Study  plan  as  a  central  element  of  communication  and 

coordination: 

•  guides  study  execution; 

•  documents  study  planning  assumptions; 

•  documents  study  planning  decisions  regarding 
alternatives  chosen; 

•  facilitates  communication  among  stakeholders; 

•  organises  and  ensures  communication  between  study 
and  client  teams; 

•  drafts  a  staged  study  plan; 

•  should  document  deviations  from  the  original  plan  and 
give  reasons  why; 

•  provides  a  baseline  for  progress  measurement  and 
study  control. 


This  phase  requires  the  ability  of  the  study  team  to  capture  and  inteipret  the  clients’  knowledge  and  expertise 
as  well  as  their  attitude  towards  existing  solutions  and  their  effective  involvement  into  the  decision-making 
process  [12].  This  process  may  involve  the  conduct  of  interviews  with  the  client(s).  The  results  of  these 
interviews  may  be  visually  presented  as  individual  cognitive  maps  that  will  later  on  converge  into  a  group 
map  and  will  constitute  the  starting  point  for  the  development  of  a  common  understanding  of  the  problem 
environment  [12]. 

Another  major  aspect  within  this  phase  is  the  recognition  of  the  nature,  level  and  source  of  uncertainty; 
the  decision  on  how  to  cope  with  the  identified  type  of  uncertainty  is  depicted  in  Figure  2-2  and  affects  the 
further  study  design  in  terms  of,  for  example,  choice  of  methodology.  The  basic  approaches  to  dealing 
with  (respectively  prepare  for)  uncertainty  are  discussed  in  Section  2.3.  Section  6.3  of  this  CoBP  discusses 
approaches  for  the  analysis  and  handling  of  uncertainty  in  detail. 

At  the  end  of  the  Appreciation  phase  the  study  team  and  the  clients  are  expected  to  have  a  common 
understanding  of  the  problem  environment  and  of  the  next  steps  to  undertake.  This  should  be  documented 
in  the  study  plan.  Furthermore,  the  study  team  will  not  only  have  a  clear  understanding  of  the  structure  of 
the  client  system  and  the  stakeholders  that  comprise  it,  but  will  also  have  a  strategy  for  coping  with 
different  types  of  stakeholders. 


5.3.2  The  Analysis  Phase 

After  exploring  the  problem’s  environment  and  the  outline  of  the  problematic  situation  in  the  Appreciation 
phase,  the  problem  itself  moves  to  the  centre  of  attention  within  the  Analysis  phase. 

For  this  reason  the  major  work  of  the  study  team  consists  of  the  following  tasks: 

•  Analysis  of  correlation  and  causal  structure; 

•  Selection  of  appropriate  (multi-)methodology; 

•  Development  of  model(s); 

•  Information/Data  gathering7;  and 

•  Generation  of  model  results. 

7  The  ‘Development  of  Model(s)’  task  may  be  separated  from  the  ‘Infomiation/Data  Gathering’  task  but  this  is  not  always 
necessary:  e.g.  drafting  a  diagram  of  all  issues  that  define  the  problematic  situation  is  in  fact  the  development  of  a  model  and 
organisation  of  information  at  the  same  time. 
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For  a  successful  conduct  of  the  Analysis  phase  a  close  interaction  with  (part  of)  the  clients  is  necessary. 
Therefore,  the  clients  should  provide  time  for  the  following  tasks: 

•  Support  with  information/data;  and 

•  Review  and  refinement  of  study  focus. 

As  a  central  result  of  this  phase,  the  study  team  and  the  clients  have  achieved  and  agreed  to  a  common 
problem  understanding  that  will  be  the  basis  for  the  selection  of  an  appropriate  (multi-)  methodology. 

After  reaching  a  common  understanding  of  the  nature  of  the  problem,  the  study  team  faces  the  task  of 
defining  an  appropriate  (multi-)  methodology  and  with  it  the  gathering  of  necessary  information  and  data. 
This  may  take  place  during  workshops  conducted  with  (parts  of)  the  (group  of)  client(s). 

The  amount  of  data  and  information  accumulating  while  developing  and  processing  some  specific  model 
approaches  can  be  large  and  may  at  first  appear  unmanageable  to  the  participants  involved  in  the  decision¬ 
making  process.  In  these  cases  an  approach  to  resolve  this  situation  is  to  develop  a  road  map  that  depicts 
the  achievements  reached  so  far  as  well  as  their  interrelations  [2].  Besides  the  improvement  of  the  problem 
understanding,  this  approach  supports  the  handling  of  the  complexity  of  the  problematic  situation  even  if  it 
cannot  reduce  it.  It  should  be  noted  that  reduction  in  complexity  may  negatively  influence  the 
effectiveness  of  the  action  plan  chosen  as  a  result  of  the  iterative  process. 

Such  roadmaps  may  not  be  easily  understandable  by  those  not  directly  involved  in  the  study.  Ownership 
of  them  is  best  allocated  those  who  were  closely  involved  in  their  creation  [2],  It  is  therefore  strongly 
recommended  that  its  creation  should  take  place  in  a  facilitated  workshop  (ref.  Chapter  7  on  workshop 
facilitation)  which  equally  involves  the  study  team  as  well  as  (parts  of)  the  (group  of)  client(s). 
This  furthermore  is  a  way  to  involve  the  client(s)  in  the  model  building  process  and  to  ensure  a  common 
problem  understanding. 

A  crucial  element  in  the  choice  of  methodology  as  well  as  the  modelling  process  itself  is  the  handling  of 
uncertainty  factors  which  have  been  identified  in  the  Appreciation  phase.  The  impact  of  uncertainty 
attached  to  the  data  and  information  gathered  in  this  phase  has  to  be  mitigated  by  choosing  an  appropriate 
methodology  (ref.  Section  6.3). 

The  validation  of  judgement-based  models  is  rather  of  a  qualitative  nature.  Particularly  Chapters  2,  3  and  7 
deal  with  issues  of  validation  and  verification.  One  important  aspect  of  conducting  ‘soft’  OA  according  to 
Phillips  [13]  (see  also  Section  5.4)  lies  in  ensuring  that  everyone  involved  in  the  decision-making  process 
approves  of  the  methodological  approach  chosen  and  the  respective  model  developed.  Therefore,  it  is 
inevitable  that  the  study  team  works  closely  together  with  the  (group  of)  client(s)  in  a  controlled  and 
facilitated  manner. 

5.3.3  The  Assessment  Phase 

Having  generated  a  set  of  model  results,  the  study  team  has  to  assess  these  results  and  generate  action 
alternatives  in  regard  to  the  shared  understanding  of  the  problem  as  well  as  the  previously  specified  and 
agreed  study  focus.  For  this,  the  Assessment  phase  consists  of  the  following  tasks: 

•  Assessment  of  model  results; 

•  Generation  of  options; 

•  Exploration  of  option  impacts; 

•  Assessment  of  options;  and 

•  Evaluation  and  recommendation. 
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The  assessment  of  the  model  results  may  imply  a  variety  of  lines  of  action.  The  generation  and  assessment 
of  these  options  and  their  impact  is  a  major  task  for  the  study  team  within  this  phase.  Furthermore, 
the  effects  of  the  choice  of  how  to  deal  with  the  factors  of  uncertainty  identified  in  the  first  phase  and 
considered  in  the  model  development  of  the  second  phase  manifest  themselves  in  the  Assessment  phase. 

Becoming  aware  in  this  phase  that  the  approach  to  coping  with  the  type  of  uncertainty  initially  assumed  to 
belong  to  the  problematic  situation  has  been  the  wrong  one,  could  necessitate  the  repetition  of  the  second 
(reconfiguring  the  methodological  approach)  or  even  the  first  (reconsidering  the  initial  assumptions) 
phase. 

After  defining  and  evaluating  a  set  of  action  options,  the  study  team  identifies  a  recommended  sub-set  of 
these  options.  Consequently,  the  (group  of)  client(s)  has  to  fulfil  the  task  ‘Prioritisation  of  options’. 

As  a  result,  a  rated  (sub-)  set  of  options  that  may  resolve  (part  of)  the  problematic  situation  is  defined  and 
builds  the  starting  point  for  the  following  Action  phase. 

5.3.4  The  Action  Phase 

The  Action  phase  begins  with  the  transformation  of  the  set  of  prioritised  options  into  a  (number  of)  action 
plan(s)  which  is  subsequently  followed  by  the  implementation  of  a  chosen/agreed  action  plan.  This  task 
itself  will  involve  the  analysis  and  assessment  of  alternative  action  plans  which  may  be  organised  similarly 
to  the  first  two  ‘A-phases’  of  this  chapter  and  which  may  require  a  small  study  on  its  own. 

The  study  team  could  offer  the  following  types  of  support  to  the  client(s): 

•  Support  of  generation  of  action  plan; 

•  Support  of  implementation  of  action  plan;  and 

•  Support  of  monitoring  of  action  plan  effects. 

The  following  tasks  have  to  be  carried  out  by  the  (group  of)  client(s)  in  the  Action  phase,  possibly  supported 
by  the  study  team: 

•  Generation  of  and  decision  on  action  plan;  and 

•  Implementation  and  monitoring  of  action  plan. 

In  a  facilitated  meeting,  the  study  team  together  with  the  client(s)  identify  an  action  plan  that  is  most  likely 
to  meet  the  requirements  that  have  been  specified  throughout  the  iterative  ‘soft’  OA  study  process. 
Subsequently,  the  action  plan  is  implemented  by  the  client  organisation. 

The  implementation  of  the  chosen  action  plan  can  produce  effects  that  may  not  have  been  predicted  or  that 
may  hint  to  a  false  analysis  and  assessment  of  the  current  problem  and  its  environment.  It  may  therefore  be 
necessary  to  reconsider  earlier  phases  (or  tasks)  of  the  iterative  ‘soft’  OA  study  process  or  to  repeat  it 
entirely  starting  with  the  Appreciation  phase. 


5.4  ITERATIONS  THROUGH  THE  ‘SOFT’  OA  STUDY  PROCESS 

The  application  of  mixed  methods/methodologies  is  quite  often  the  most  promising  way  of  meeting  the 
demands  of  the  multi-faceted  nature  of  problematic  situations.  In  addition,  one  or  more  of  the  four 
A-phases  may  have  to  be  conducted  more  than  once.  Furthermore,  this  may  take  place  sequentially  or  in 
parallel  -  where  indicated  and  possible  -  and  may  lead  to  various  combinations  of  iteration  of  ‘soft’  OA 
techniques. 
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Figure  5-3  gives  an  example  of  a  sequential  iteration  through  the  ‘soft’  OA  study  process.  Whereas  earlier 
in  this  chapter  the  possibility  of  jumping  back  to  preceding  phases  with  respect  to  the  application  of  a 
single  methodology  has  already  been  indicated,  Figure  5-3a  shows  the  adaptation  of  this  principle  for  two 
methods  (one  for  each  cycle)  and  Figure  5-3b  shows  that  methods  may  be  different  per  phase  per  cycle. 


Figure  5-3a:  Example  of  Sequential  Phase  Iteration  (A  Different  Method  Per  Cycle). 


Figure  5-3b:  Example  of  Sequential  Phase  Iteration  (A  Different  Method  Per  Phase  Per  Cycle). 

If,  however,  for  a  specific  problematic  situation  the  application  of  two  methodologies  (e.g.  mixed 
‘hard’/’soflt’  or  ‘soft’/’soft’)  may  seem  appropriate,  a  possible  iteration  may  induce  the  parallel  conduct  of 
the  Analysis  and  Assessment  phases  as  depicted  in  Figure  5-4.  Such  possibilities  should  be  taken  into 
consideration  when  designing  the  study  during  the  Appreciation  phase. 


Figure  5-4:  Example  of  Partially  Parallel  Phase  Processing. 


It  may  be  necessary  to  run  through  all  phases  for  each  methodology,  leading  to  a  parallel  two-threaded 
‘soft’  OA  study  process  as  depicted  in  Figure  5-5. 


Method  A 


Method  D 


Method  B 


Method  C 


Method  E 


Method  F 


Figure  5-5:  Example  of  Full  Parallel  Phase  Processing  with  Different  Methods. 
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The  nature  of  the  specific  problematic  situation,  as  well  as  the  evaluation  of  the  (group  of)  client(s), 
will  help  determine  whether  this  setting  should  be  dealt  with  in  two  separate  studies  or,  if  coherence 
requires,  they  are  best  treated  by  parallel  execution.  Chapter  6  discusses  in  detail  the  multi-methodological 
approach  in  the  context  of  the  application  of  ‘soft’  OA. 

The  four  A-phases  of  the  ‘soft’  OA  study  process  describe  the  predominant  activity  in  terms  of  the  ‘global’ 
(or  higher)  problem  level  whereas  at  a  ‘local’  (or  lower)  problem  level  (respectively  at  one  smaller  sub-set  of 
the  problematic  situation)  all  four  activities  are  conducted  within  the  respective  main  phase  before  reaching 
the  next  main  phase. 

Figure  5-6  illustrates8  how  each  main  phase  may  include  tasks  (and  methods)  that  are  typical  of  other  main 
phases,  in  a  varying  intensity  and  most  likely  requiring  a  varying  amount  of  time.  Of  course,  at  the  global 
study  level,  the  major  emphasis  will  be  on  the  tasks  and  the  use  of  methods  that  are  typical  of  a  specific 
main  phase  (i.e.  the  diagonal  from  the  lower  left  comer  to  the  upper  right  comer  of  the  diagram),  but  at  the 
local  study  level  within  a  main  phase  there  may  be  a  similar  sequence  of  study  phases.  Consideration  of 
the  other  phases  will  also  ensure  that  the  overall  study  goal  is  kept  in  mind  while  working  at  the  specific 
phase. 
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Figure  5-6:  Each  Main  Phase  Requires,  in  a  Varying 
Intensity,  Tasks  that  are  Typical  of  All  Phases. 


An  example  within  the  Analysis  phase  may  look  like  this: 

*  Appreciation  of  the  problem’s  defining  factors,  including  the  selection  of  a  suitable  methodological 
approach  for  problem  analysis  (medium  intensity  in  Figure  5-6). 


See  [14]  for  an  alternative  way  of  illustrating  this. 
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•  Analysis  of  the  problem’s  defining  factors  and  their  interrelationships  (perhaps  cause-effect 
relations)  by  developing  and  using  a  (causal)  model  and  generating  model  results  (high  intensity 
in  Figure  5-6). 

•  Assessment  of  the  results  (key  concepts,  key  objectives,  etc.)  and  the  initially  chosen  methodological 
approach  for  the  study  (medium  intensity  in  Figure  5-6). 

•  Action  A:  restart  overall  Analysis  phase  with  different  approach,  or  Action  B:  proceed  to  overall 
Assessment  phase  (low  intensity  in  Figure  5-6). 


In  Figure  5-2,  this  correlation  is  depicted  by  a  series  of  small-sized 
A-phases  at  a  secondary  level  within  a  main  phase  (Figure  5-7). 


Appr.  Ana.  Asses.  Act. 

Figure  5-7:  Sequence  of  A-phases 
at  the  Secondary  Level. 


A  ‘hard’  modelling  approach  often  has  a  natural  point  where  one  can  decide  on  its  ‘readiness’  for  use: 
a  thorough  verification  (and  validation)  procedure  will  usually  suffice.  However,  this  may  not  be  at  all 
obvious  when  constructing  judgement-based  models.  A  judgement  cannot  easily  be  verified  as  ‘correct’. 

Phillips  (e.g.  [13])  introduced  the  useful  concept  of  a  ‘ requisite  ’  model  which,  assuming  that  it  will  serve 
as  an  aid  to  thinking  and  group  learning,  is  sufficient  in  form  and  content  to  resolve  the  issues  at  hand  in 
the  opinion  of  the  group  of  people  who  construct  the  model.  Therefore,  the  model  represents  their 
collective  view  and  shared  understanding  at  that  stage  of  the  process. 

The  process  of  identifying  a  way  ahead  when  a  problematic  situation  has  presented  itself  may  be  a  matter 
of  debate  among  a  group  of  clients  and  other  stakeholders.  A  requisite  way  ahead  is  one  that  is  agreed  by 
the  group  to  be  feasible  and  to  sufficiently  offer  perspectives  of  improving  or  resolving  that  situation. 
The  aim  of  the  study  may  require  a  certain  nature,  precision  or  quality  of  data  which  may  not  be 
achievable  with  reasonable  effort  and  costs.  Again,  an  iterative  process  of  reviewing  and  adjusting  the 
initially  chosen  model  type  and  method,  and  perhaps  even  the  study’s  aim,  can  lead  to  a  requisite  approach 
based  on  debate  and  negotiation. 

When  building  a  model  as  a  construct  of  concepts  and  relationships  or  as  a  criterion  hierarchy  for  evaluation 
purposes  or  any  other  type  of  model  based  on  judgement,  the  group  will  end  the  modelling  process  when  it 
feels  that  ‘requisiteness’  has  been  achieved.  That  point  has  been  reached  when  everybody  feels  comfortable 
with  the  model. 


5.5  DIVERGENT  AND  CONVERGENT  THINKING 

The  effect  of  opening  up  the  problem  space  is  referred  to  as  divergence  and  describes  the  increase  in 
complexity  through  adding  new  aspects  to  the  problem. 

‘Divergence’  is  more  often  experienced  in  ‘soft’  OA  rather  than  in  ‘hard’  OA  studies  (Figure  5-8).  The  clear 
problem  description  typical  of  applications  of  ‘hard’  methods  implies  that  a  divergent  stage  is  not  needed. 
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Divergent  study  Convergent  study 

Figure  5-8:  Comparison  of  Divergence  and  Convergence  in  ‘Soft’  and  ‘Hard’  OA  Studies. 

‘Divergence’  opens  up  the  problem  space  which  quite  often  may  be  necessary  and  useful  in  order  to  let  the 
‘real’  problem  emerge.  Consequently,  one  of  the  major  tasks  of  an  experienced  study  team  is  to  balance 
diverging  and  converging  streams  of  thoughts  throughout  the  ‘soft’  OA  study  process  in  order  to  reach  a 
solution  that  addresses  the  right  problem  (or  problematic  situation)  and  that  is  understood  and  accepted  by 
all  stakeholders. 

Divergence  can  appear  in  all  phases  of  the  process  as  a  result  of  discussions  (or  brainstorming)  between 
different  people  with  different  opinions  (or  agendas)  and  will  increase  complexity  and  turnaround  time  of 
the  process.  On  the  other  hand,  it  may  be  necessary  in  order  to  identify  the  ‘real’  problem  (or  problematic 
situation)  or  to  find  the  best  possible  solution  or  management  approach  to  it. 

Figure  5-9  depicts  a  cycle  of  divergent  and  convergent  thinking.  This  figure  illustrates  that  convergent 
lines  of  thinking  may  not  always  reach  a  satisficing  end  point  at  some  stage  of  the  process,  for  example 
End  Point  A,  but  rather  Point  B  or  Point  C  which  may  not  be  generally  acceptable.  In  these  cases  iteration 
enables  the  rethinking  of  the  current  approach  and  provides  an  opportunity  to  get  back  ‘on  track’  towards 
an  end  point  that  is  more  satisfactory. 
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Figure  5-9:  Iteration  and  Convergent  Thinking. 


Typical  activities  during  stages  of  divergent  and  convergent  thinking  are  given  in  Table  5-1. 


Table  5-1:  Divergent  versus  Convergent  Thinking  (taken  from  [15]). 


Divergent  Thinking 

Convergent  Thinking 

Generating  a  list  of  ideas 

VS. 

Sorting  ideas  into  categories 

Free-flowing  open  discussion 

vs. 

Summarising  key  points 

Seeking  diverse  points  of  view 

vs. 

Coming  to  agreement 

Suspending  judgement 

vs. 

Exercising  judgement 

An  experienced  facilitating  analyst  will  encourage  streams  (or  stages)  of  divergent  and  convergent 
thinking  throughout  the  whole  process  in  order  to  identify  the  most  acceptable  way  ahead.  The  reader  is 
referred  to  [15]  fore  more  aspects  of  divergence  and  convergence  in  the  context  of  working  with  groups. 
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Chapter  6  -  DESIGNING  THE  ‘SOFT’  OA  STUDY  METHODOLOGY 


•  Chapter  1  -  INTRODUCTION 

•  Chapter  2  ■  PROBLEMATIC  SITUATIONS  AND  ‘SOFT’  OA 

•  Chapter  3  ■  ACHIEVING  VALIDITY,  CREDIBILITY  AND  ACCEPTANCE 


Chapters  ROLES  AND  RESPONSIBILITIES 


•  Chapter  5  ■  THE  'SOFT  OA  STUDY  PROCESS 

•  Chapters-  DESIGNING  THE  'SOFTOA  STUDY  METHODOLOGY 

•  Chapter  7  ■  DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 

•  Chapter  8  ■  INTERPRETING,  USING  AND  COMMUNICATING  RESULTS  AND  OUTCOMES 


•  Chapter  9  ■  THE  ESSENCE  OF  THE  CODE  OF  BEST  PRACTI CE 

•  Annex  A  -  LIST  OF  KEY  STATEMENTS  AND  RECOMMENDATIONS  OF  IMPORTANCE 

TO  THE  ANALYST 

•  Annex  B  •  SELECTIVE  LIST  OF  'SOFT'  METHODS 


>  Communicate  with  client(s)  including  key  stakeholders  and  seek  agreement  on  any  assumption, 
action  and  delivery. 

>  Determine  the  most  likely  nature  of  the  problematic  situation  by  trying  to  shed  light  on  key  aspects. 

>  Decide  on  the  appropriateness  of  a  trajectory  from  perceived  mess  to  either  a  problem  proper  or  a 
manageable  mess. 

>  Identify  which  type(s)  of  uncertainty  exist(s)  and  contemplate  on  both  the  adequate  attitude  to 
uncertainty  and  ways  of  coping  with  it. 

>  Identify  stakeholders  and  their  interests  and  attitudes  towards  the  study  by  conducting  a  stakeholder 
analysis. 

>  Decide  on  the  appropriateness  of  a  single  or  a  multi-methodology  study  approach. 

>  Examine  all  methods  on  their  merits  to  the  problematic  situation  at  hand  and  decide  on  the 
appropriateness  of  specific  candidate  methods.  Recognise  own  limitations  in  knowledge/expertise  and 
seek  assistance. 

>  Decide  on  data  and  other  resources  needed  including  the  use  of  software. 

>  Be  aware  of  the  danger  of  dissipation  of  results  from  ‘hard’  OA  methods  when  interleaving  ‘hard’  and 
‘soft’  OA  methods  in  a  multi-methodology  approach. 

>  Maximise  objectivity  and  rigour,  gauging  the  validity  of  the  overall  approach  in  terms  appropriate  to  the 
problem  domain  being  addressed. 

>  Document  the  argumentation  and  rationale  for  modelling  decisions  and  keep  a  record. 

>  Document  the  argumentation  and  rationale  for  changes  in  the  study’s  assumptions  and  aim  (including 
aims  of  specific  study  phases),  the  definition  (or,  alternatively,  the  common  understanding)  of  the 
problematic  situation  in  order  to  ensure  end-to-end  integrity. 
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6.1  INTRODUCTION 

This  chapter  addresses  the  design  of  the  methodological  approach  of  a  study  and  in  particular,  whether  and 
how  different  methods  may  be  combined.  As  already  stated  in  Chapter  1,  the  CoBP  will  not  discuss 
specific  methods  but  this  chapter  will  provide  general  guidance  to  deciding  on  the  choice  of  an  appropriate 
study  methodology  and  will  give  some  examples  of  appropriate  methods. 

This  chapter  starts  with  guidelines  as  to  how  to  determine  the  nature  of  the  problematic  situation  since  this 
will  affect  the  methods  and  methodologies  to  be  used.  After  that  the  chapter  addresses  the  phenomenon  of 
uncertainty  in  more  depth  than  Chapter  2  by  suggesting  a  typology  which  may  be  useful  when  designing  the 
study  methodology.  The  chapter  then  discusses  stakeholder  analysis  and  mapping  as  a  follow-up  on  Chapter 
4’s  discussion  of  the  analyst’s  perspective  on  the  stakeholders.  The  chapter  goes  on  to  address  the  modelling 
process  and  differentiates  between  a  study  approach  based  on  a  single  methodology  (usually  for  a  small- 
scale  study)  and  one  that  is  based  on  multiple  methods  (usually  for  larger-scale  studies).  Collective 
experience  (see  e.g.  [1])  has  shown  that  most  of  the  time,  there  is  a  need  for  a  multi-methodology  approach 
when  addressing  a  mess.  Much  of  what  will  be  suggested  regarding  single-methodology  studies  - 
for  example  about  choosing  and  designing  a  study  methodology,  about  modelling  issues,  data  requirements, 
use  of  experts,  etc.  -  has  a  general  validity  and  will  therefore  be  valid  for  multi-methodology  studies  as  well. 

6.2  HOW  TO  COPE  WITH  PROBLEMATIC  SITUATIONS  AND  RECOGNISE 
MESSES 

When  an  analyst  is  called  in  for  support,  one  of  the  first  matters  he  has  to  consider  is  the  nature  of  the 
problematic  situation  and  the  type  of  analysis  that  will  be  most  appropriate. 

Chapter  2  discusses  three  fundamental  problem  types  that  represent  a  broad  spectrum  of  problematic 
situations:  puzzles,  problems  (proper)  and  messes.  At  the  outset  of  a  study  it  may  not  be  clear  which  type 
it  is.  Communication  with  the  client(s)  whilst  referring  to  the  main  characteristics  of  the  three  problem 
types  and  to  those  listed  in  Table  2-2  should  make  matters  clearer. 

Below  follows  a  step-wise  approach  which  may  help  the  analyst  determine  which  type  of  problematic 
situation  he  is  facing,  before  designing  the  study  methodology:  ask  questions,  follow  a  checklist,  interpret 
the  exploratory  findings  so  far,  decide  on  an  additional  trajectory  to  get  things  clearer  in  the  case  of  a 
tentatively  perceived  mess. 

The  analyst  must  be  able  to  discuss  the  client’s  initial  statements  (which  could  be  ill-perceived)  and  identify 
which  type  should  be  assumed  to  be  initially  applicable.  This  could  be  done  at  an  early  stage  of  his 
involvement  by  asking  questions  regarding  at  least  the  following  issues: 

•  Symptoms,  possible  causes,  history 
(and:  “is  this  all  there  is?”:  initially 
hidden  problem  behind  stated  problem); 

•  Identification  of  stakeholders,  and  their  backgrounds,  interests,  aims,  and  anticipated  commitment; 
(“who  is  unhappy  with  the  problem  and  why?”,  “who  does  and  who  does  not  want  the  problem  to  be 
resolved  and  why?”,  “who  will  and  who  will  not  be  cooperating?”,  “when  can  the  problematic 
situation  be  considered  as  ‘improved’?”); 

•  Uncertainties  about  the  previous  issues  and  resources  needed  and  anticipated  consequences  of 
(not)  addressing  the  problem,  fear  of  these  consequences,  restrictions  (and:  “imposed  by  whom?”); 

•  The  relationship  between  the  ultimate  decision  maker  and  the  problem  owner  (and:  “is  there 
something  to  decide  upon?”,  “is  ‘do  nothing’  a  feasible  option?”,  “why  is  this  your  problem?”); 


Anonymous  client  quotation: 

“I  want  to  exclude  unforeseen  outcomes” 
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•  Any  deadlines,  the  time  frame  within  which  first  deliverables  are  expected  (and:  “how  urgent  is 
this?”);  and 

•  Payment  and  other  administrative  matters  which  however  will  serve  as  a  means  for  the  analyst  to 
gain  more  insight  into  internal  client  relations  (“who  is  paying?”,  “same  person  as  problem  owner?”, 
“who  is  really  willing  to  spend  money”?). 

When  analysing  the  answers  to  the  questions  above,  the  following  checklist  (based  on  Chapter  2’s  Table 
2-2  comparing  ‘soft’  and  ‘hard’  OA  and  its  cited  references,  and  also  [2],  [3])  will  help  the  analyst  determine 
if  he  is  dealing  with  a  messy  situation,  a  problem  (proper)  or  merely  a  puzzle: 

□  Not  much  is  initially  known  about  the  nature  of  the  problematic  situation  and  its  boundaries 
(what  matters  and  what  does  not). 

□  Not  much  is  initially  known  about  defining  the  elements  of  the  problematic  situation  and  how  they 
may  be  interrelated. 

□  Not  much  is  initially  known  about  who  the  stakeholders  are  and  in  what  manner  they  may  be 
directly  or  indirectly  affected,  their  viewpoints  and  what  they  are  worried  about. 

□  Not  much  is  initially  known  about  the  goals,  objectives  and  measures  of  effectiveness  or  merit  that 
may  be  relevant. 

□  Not  much  is  initially  known  about  what  can  and  should  be  changed  towards  improvement  of  the 
problematic  situation,  and  under  what  conditions  or  according  to  what  criteria  a  change  will  be 
regarded  as  an  improvement. 

□  Not  much  is  initially  known  about  the  data  needed,  its  relevance,  availability  and  reliability. 

□  Not  much  is  initially  known  about  the  way  in  which  changes  in  context  will  affect  the  problematic 
situation,  its  improvement  and  the  study  design  to  achieve  it. 

□  Power,  emotion,  politics  and  ethics  will  most  likely  come  into  play,  but  not  much  is  initially  known 
about  how  and  to  what  effect. 

□  Different  people  say  different  things  (or  express  different  views)  about  the  same  issue. 

□  Not  much  is  known  about  key  interactions  of  human  cognition,  beliefs  and  behaviour. 

□  Not  much  is  known  about  the  context  of  grand-scale  issues  that  have  no  clear  end-points,  transcend 
specific  domains,  and  have  significant  political  or  public  policy  implications  (e.g.  counter-terrorism). 

□  Not  much  is  known  about  where  any  possible  resolution  will  most  likely  have  side-effects  attached 
to  them  that  are  undesirable  by  stakeholders. 

The  ‘not  much  is  (initially)  known’  part  of  the  list  above  denotes  that  a  large  amount  of  uncertainty  may 
exist:  uncertainty  about  different  aspects  of  a  problematic  situation  and  different  types  and  levels  of 
uncertainty.  This  will  be  discussed  in  Section  6.3. 

The  problematic  situation  may  be  regarded  as  a  puzzle  by  the  analyst  if  (virtually)  none  of  the  previous 
checklist  items  are  ticked  and  if  a  search  is  required  for  a  quantitatively  undeipinned  outcome,  preferably  the 
best  possible  (the  optimum),  to  inform  the  decision  to  be  made.  Some  of  the  items  listed  in  the  checklist  may 
require  ‘soft’  methods  to  further  delineate  and  define  them.  For  example,  there  may  be  a  need  to  interview 
stakeholders  to  agree  the  formulation  of  the  model  and  the  nature  of  the  required  solution,  but  this  will  be 
followed  by  a  ‘hard’  modelling  exercise  in  case  of  a  puzzle. 

The  problematic  situation  may  be  regarded  as  a  problem  (proper)  by  the  analyst  if  only  a  few  of  the 
previous  items  are  ticked  and  there  is  a  feeling  amongst  both  the  analyst  and  the  client  feel  that  agreement 
can  be  achieved  in  the  formulation  of  the  problem,  the  definition  and  (semi-)quantitative  evaluation  of  the 
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potential  solution  options,  and  the  procedure  to  achieve  results.  The  study  methodology  will  in  this  case 
most  likely  consist  of  a  combination  of ‘hard’  and  ‘soft’  methods. 

If  most  of  the  previous  items  are  ticked  the 
analyst  will  be  wise  to  regard  the  problematic 
situation  as  a  mess.  In  that  case  the  CoBP 
suggests  two  ways  forward:  create  a  problem 
(proper)  (i.e.  a  ‘type  1  trajectory’),  or,  if  that 
fails,  continue  to  work  in  the  mess  space  (i.e.  a 
‘type  2  trajectory’).  The  two  trajectories  are 
discussed  next. 

A  type  1  trajectory  (perceived  mess  problem 
proper)  should  aim  for  a  common  agreement 
among  the  primary  client  and  key  stakeholders, 
before  actually  conducting  the  analysis  proper, 
on  at  least: 

•  The  formulation  of  the  problem  (creating  a  well-defined  construct)  one  can  work  with; 

•  Setting  initial  restrictions1  in  physical,  organisational,  social,  time-related,  etc.,  terms  as  they  could 
affect  the  analysis  approach; 

•  The  nature  of  the  options  and  the  ability  to  (semi-)quantitatively  evaluate  and  compare  them; 

•  The  type(s)  of  model  to  be  constructed  and  used,  and  the  desired  nature  of  results  and  the  outcome 
of  the  study  as  a  whole; 

•  Participants  in  the  study  (particularly  in  respect  of  specific  types  of  expertise)  and  the  information 
each  individual  will  bring  in  (or  give  access  to);  and 

•  The  internal  phasing  procedure  of  the  study  needed  to  achieve  results. 

A  type  2  trajectory  (perceived  mess  manageable  mess)  should  aim  for  a  common  desire,  an  initial 
commitment  among  primary  client  and  key  stakeholders,  that  something  can  and  should  be  done  to 
understand  and  change  the  current  problematic  situation.  Along  the  way  a  clearer  picture  should  be  obtained 
of  some  key  elements  of  the  anticipated  study,  and  thus  turn  the  perceived  mess  into  a  mess  that  one  can  start 
working  on: 

•  An  identification  and  better  understanding  of  aspects  of  the  problem  and  the  factors  that  influence  it; 

•  The  problematic  aspects  to  be  addressed  and  improved; 

•  A  view  of  any  boundaries  to  be  set  initially; 

•  The  nature  of  the  anticipated  outcome  of  the  study; 

•  Participants  in  the  study  (particularly  in  respect  of  specific  types  of  expertise)  and  the  information 
each  individual  will  bring  in  (or  may  give  access  to);  and 

•  An  initial  design  of  the  internal  phasing  procedure  of  the  study  needed  to  achieve  results. 

Trajectory  1  could  be  a  study  in  itself,  the  end  of  which  could  lead  to  the  start  of  a  problem  (proper) 
solving  study.  Trajectory  2  will  most  likely  be  undistinguishable  from  the  actual  study,  although  one  or 
two  moments  of  reflection  and  a  go/no-go  decision  could  be  built  in. 


1  The  definition  of  restrictions  (i.e.  boundaries  to  the  study)  at  some  stage  of  the  study  is  important:  how  far  to  go  back  in 
history,  for  example,  may  determine  whether  or  not  some  of  the  root  causes  of  a  conflict  will  be  addressed. 


A  new  combat  boot  had  been  introduced  in  Sweden  when 
the  number  of  foot  injuries  started  rising  alarmingly.  The 
issue  identified  during  the  workshops  using  Morphological 
Analysis  was  a  lack  of  knowledge  of  the  range  of  injuries 
that  had  been  sustained.  A  survey  was  conducted  in  order 
to  map  the  issue.  The  survey  was  not  part  of  the  original 
study  design,  but  an  outcome  of  the  acute  lack  of 
information  the  working  group  experienced.  The 
morphological  model  was  used  the  help  shape  the 
questions  in  the  survey.  Even  though  the  study  was  not 
finalised  as  originally  planned,  the  application  of 
Morphological  Analysis  was  valuable  by  transforming  the 
combat  boot  issue  from  a  mess  to  a  problem,  and  by 
exposing  the  lack  of  data. 
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Figure  6-1  illustrates  the  trajectory  methodology  for  coping  with  problematic  situations  discussed  above. 
A  large  part  of  it  could  be  canned  out  in  a  facilitated  workshop  setting  (ref.  Chapter  7  on  workshop 
facilitation)  with  key  actors.  Part  of  this  process  may  be  iterative  (although  not  shown  in  the  diagram)  until 
a  common  understanding  and  agreement  on  the  way  ahead  is  achieved2. 


Recall  from  Chapter  4  that  the  analyst  very  often  has  to  ‘prove’  his  competence  in  the  methods  and 
methodologies  of  ‘soft’  OA  when  a  client’s  initial  attitude  is  that  of  regarding  the  analyst  as  a  technical 
scientist  familiar  with  traditionally  ‘hard’  analytical  work.  Adopting  one  of  the  two  trajectories  may  be  a 
way  to  show  his  skills  at  an  early  stage. 


6.3  ANALYSING  AND  COPING  WITH  UNCERTAINTY 

Uncertainty  is  a  phenomenon  that  has  to  be  coped  with  in  any  operational  analysis  as  it  is  often  the  prime 
characteristic  of  the  problematic  situation.  Chapter  2  discusses  the  concept  of  uncertainty  in  a  general 
sense  and  suggests  three  different  attitudes  that  one  might  adopt  towards  uncertainty.  Each  of  them  leads 
to  some  specific  form  of  preparation  for  uncertain  events.  It  is  suggested  in  Chapter  2  that  for  coping  with 
uncertainty  related  to  the  type  of  (messy)  problems  the  CoBP  is  addressing,  a  mix  of  those  attitudes  will 
be  most  recommendable. 

Before  discussing  the  modelling  process  itself  this  section  will  discuss  the  various  types  of  uncertainty  in 
more  depth  with  a  view  to  the  modelling  consequences.  This  typology  helps  the  analyst  with  identifying 
the  nature  of  uncertainty  that  exists  in  the  problematic  situation  he  is  dealing  with  and  therefore  with 


2 

The  concept  of  ‘requisiteness’  is  usually  applied  to  the  modelling  process  (ref.  Section  5.4),  but  it  is  also  relevant  to 
identifying  a  way  ahead  that  is  sufficient  in  form  and  content  to  resolve  the  issues  at  hand  in  the  opinion  of  the  group  of 
people  who  will  be  involved  in  the  study  process. 
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determining  which  study  methodology  will  be  appropriate.  At  the  end  of  this  section  some  advice  will  be 
given  regarding  methods  that  are  appropriate  for  certain  types  of  uncertainty. 

Much  of  what  follows  is  based  on  Walker  et  al.  [4]  and  Dreborg  [5],  which  in  turn  are  based  on  a  large  body 
of  literature  on  uncertainty.  That  material  has  been  adapted  and  extended  for  the  purpose  of  the  CoBP. 

The  list  below  constitutes  a  typology  of  uncertainty.  A  distinction  can  be  made  between  the  following 
dimensions  and  aspects. 

•  The  locations  of  uncertainty;  uncertainty  may  manifest  itself  in  a  number  of  elements  of  the 
modelling  process: 

•  Model  context,  which  refers  to  the  framing  of  the  problem,  the  definition  of  problem  boundaries, 
the  identification  of  clients  and  other  stakeholders  and  their  perspectives,  the  identification 
of  external  sources  of  influence  (e.g.  adversaries,  other  threatening  or  non-threatening 
circumstances); 

•  Model  structure,  which  refers  to  the  form  of  the  model,  the  way  in  which  concepts  and 
relationships  between  concepts  are  defined,  system  behaviour; 

•  Model  input  and  model  parameters,  which  refers  to  variables,  data,  values  of  parameters, 
description  (expected  behaviour)  of  external  influence  factors  and  the  current  system;  and 

•  Model  results,  which  refers  to  estimates  of  outcomes/consequences,  prediction  error. 

The  other  dimensions  that  follow  are,  in  fact,  characterisations  pertinent  to  each  of  the  uncertainty 
locations  listed  above: 

•  The  levels  of  uncertainty;  different  degrees  of  uncertainty  exist  regarding  the  location  of  uncertainty, 
ranging  from  full  knowledge  (most  likely  quantitatively  available)  via  partial  (most  likely  only  to 
some  extent  quantitatively  and  to  some  extent  qualitatively  available)  to  complete  absence  of 
knowledge: 

•  Stochastic  uncertainty,  which  derives  from  measurement  issues  such  as  imprecision  and 
sampling  error,  and  the  inherent  probabilistic  nature  of  variables  (‘games  against  mother  nature’) 
-  this  type  of  uncertainty  can  usually  be  treated  quantitatively  as  the  event  space  will  most 
likely  be  known. 

•  Scenario  uncertainty,  which  is  uncertainty  about  the  future  external  environment,  the  range  of 
possible  outcomes  as  a  result  of  uncertain  mechanisms  although  the  probability  values  are 
unknown  -  two  types  of  scenario  uncertainty  can  be  distinguished,  both  of  which  could  be 
treated  quantitatively  or  qualitatively  or  as  a  mixture  of  both: 

o  Intentional  (defence  issues  are  normally  characterised  by  the  existence  of  adversarial 
actors,  but  this  also  includes  situations  where  a  number  of  stakeholders  with  divergent 
interests  exist  having  the  means  to  exercise  influence);  and 

o  Unintentional  (trends  and  developments  to  some  extent  autonomously  leading  to  a  future 
situation). 

•  Ignorance,  which  ranges  from  ‘known  unknowns’  when  one  still  knows  what  one  does  not 
know,  to  ‘unknown  unknowns’  when  one  even  does  not  know  what  one  does  not  know  (‘black 
swans’)  [6]  -  this  type  of  uncertainty  is  usually  qualitative  and  addressing  it  relies  heavily  on 
human  judgement  and  imagination. 

•  The  nature  of  uncertainty,  which  means  that  different  sources  (or  perhaps  rather:  root  causes)  of 
uncertainty  exist  regarding  the  possible  locations  of  uncertainty: 
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•  Imperfection  of  knowledge,  which  may  be  reduced  by  more  effort  to  gain  missing  knowledge 
(e.g.  about  adversaries,  circumstances,  stakeholders)  or  by  involving  subject-matter  experts 
(who  give  infonned  estimates  or  their  knowledgeable  opinions)  or  perhaps  merely  by  the 
passing  of  time  as  more  information  presents  itself;  and 

•  Inherent  variability,  which  concerns  social,  economic  and  technological  developments  or 
natural  events  and  may  be  caused  by  the  inherent  randomness  (stochasticity)  or  dynamics  of 
processes,  technological  surprises,  and  human  behaviour  and  difference  of  opinion. 

•  The  time  dimension  of  uncertainty  which  means  that  the  level  or  nature  of  uncertainty  may  or  may 

not  change  over  time: 

•  Dynamic  uncertainty,  which  is  any  uncertainty  one  will  know  more  about  with  time,  such  as 
the  weather,  or  the  intentions  of  a  specific  adversary;  it  is  essentially  dynamic  in  relation  to 
the  degree  of  preparedness  that  exists  with  respect  to  a  certain  development,  i.e.  whether  or 
not  one  knows  in  time  what  meaningful  countermeasures  to  take;  and 

•  Static  uncertainty,  which  essentially  remains  constant  such  as  the  probability  of  an  accident 
occurring  (assuming  that  no  additional  preventive  precautions  are  taken  as  a  preparation). 

The  analyst  will  need  to  distinguish  between  the  dimensions  and  their  underlying  aspects  in  order  to 
adequately  design  methods  to  cope  with  specific  types  of  uncertainty.  Figure  6-2  shows  an  uncertainty 
matrix  (adapted  and  extended  from  Walker  et  al.  [4])  with  ‘location  of  uncertainty’  at  its  centre  (vertical 
Y-axis),  ‘levels’  and  ‘nature’  at  its  right  and  left  respectively  (horizontal  X-axis),  and  ‘time’  as  the  table’s 
third  dimension  (Z-axis).  The  items  on  the  X-axis  characterise  the  items  on  the  Y-axis.  The  items  on  the 
Z-axis  characterise  those  on  the  X-axis  for  each  of  the  location  instances  on  the  Y-axis.  The  result  is  a 
3-dimensional  matrix,  where  each  cell  may  be  elaborated  further  if  that  specific  combination  of  location 
instance,  level  type,  nature  type  and  time  aspect  exists  in  the  problematic  situation  the  analyst  is  facing. 
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Figure  6-2:  Table  of  Uncertainty  Typology  (3-dimensional)  (Extension  of  a  Table  in  [4]). 
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It  must  be  noted  that  there  are  some  relationships  between  ‘Nature’  and  ‘Level’;  for  example,  imperfection 
of  knowledge  leads  to  ignorance.  In  addition,  one  could  argue  that  ‘static  uncertainty’  is  an  extreme  case 
of  ‘dynamic  uncertainty’  as  far  as  time  is  concerned.  So,  the  dimensions  are  related  to  some  extent. 

The  purpose  of  this  table  is  to  help  the  analyst  find  ways  of  dealing  with  uncertainty  by: 

•  Identifying  appropriate  types  of  uncertainty  in  the  preparatory  pre-analysis  phase  of  a  study; 

•  Choosing  appropriate  methods  in  the  Analysis  phase  (by  examining  their  characteristics  and  relating 
these  to  the  entries  of  the  table  in  Figure  6-2)  that  are  able  to  address  identified  types  of  uncertainty; 
and 

•  Ex-post  showing  that  all  relevant  types  of  uncertainty  have  been  dealt  with  by  using  the  matrix  as 
a  quality  control  checklist. 

In  situations  of  uncertainty  where  a  quantitative  treatment  would  be  appropriate,  the  following  examples 
of  methods  would  be  useful: 

•  Risk  management  works  well  in  a  situation  that  is  characterised  by  stochastic  uncertainty,  and  can 
be  used  for  intentional  uncertainty  as  well.  Risk  management  combines  elements  of  all  three 
attitudes  towards  uncertainty  discussed  in  Chapter  2.  Risk  management  needs  to  be  supported  by 
other  approaches  in  situations  of  strong  qualitative  uncertainty. 

•  Game  theory  has  been  used  successfully  in  situations  where  intentional  uncertainty  is  strong, 
but  it  does  require  that  the  rules  of  the  game  be  known  quantitatively.  Thus  game  theory  is  less 
suited  were  intentional  uncertainty  is  combined  with  qualitative  uncertainty. 

•  Decision  trees  are  suitable  for  exploring  situations  of  dynamic  uncertainty,  but  again  assume 
mainly  quantitative  uncertainty  to  work  well. 


If  the  situation  is  characterised  by  large 
elements  of  qualitative  uncertainty,  other, 
predominantly  ‘soft’  approaches  are  called  for. 

One  important  approach  is  scenario  planning 
([7],  [8],  [9]),  which  helps  by  imagining  possible 
futures  were  different  rules  apply.  In  essence, 
scenario  planning  is  used  to  explore  bits  of  the 
event-space.  Other  methods  can  be  used  to 
similar  effect,  often  in  combination  with  scenario 
planning  (e.g.  Moipho logical  Analysis  [10]). 

In  situations  where  intentional  and  qualitative 
uncertainty  is  prevalent,  various  table  top 
(war)gaming  approaches  have  been  successful 
(e.g.  [11],  [12]).  By  assuming  the  roles  of 
potentially  adversarial  or  co-operating  actors 
one  uses  one’s  empathy  and  imagination  to 
think  of  some  of  the  novel  moves  the  real  actors 
might  apply.  A  specific  form  of  qualitative  and 
intentional  uncertainty  is  involved  with  identifying  the  possible  consequences  of  actions  one  may  take  in 
order  to  address  the  problematic  situation.  In  order  to  approach  this  issue  gaming  may  be  relevant  as  well. 

When  dealing  with  dynamic  and  qualitative  uncertainty,  one  useful  option  is  to  use  adaptive  planning 
approaches,  i.e.  devise  options  that  retain  as  much  freedom  of  action  as  possible  in  the  face  of  an  uncertain 
future.  The  Strategic  Choice  Approach  and  Robustness  Analysis  (both  in  [13])  are  helpful  in  these 
situations.  System  Dynamics,  possibly  in  combination  with  Group  Model  Building  [14],  would  be  another 


Early  2003,  the  Swedish  Army  leadership  felt  that  the  Army 
units  no  longer  had  a  unified  sense  of  purpose.  As  the 
focus  of  the  Army  had  gradually  shifted  from  homeland 
defence  towards  conducting  missions  abroad,  the  unity 
of  purpose  was  gradually  lost. 

In  order  to  address  this  challenge,  a  scenario  planning 
approach  was  chosen.  Workshops  were  assembled  where 
a  representative  selection  of  Army  personnel  was  present. 
They  were  tasked  with  brainstorming  driving  forces  that 
could  be  regarded  as  important  and  uncertain.  The  key 
question  was  “Does  Sweden  need  an  Army  in  the  future?”. 
Based  on  the  identified  driving  forces  four  scenarios  were 
constructed,  describing  four  ‘worlds’  in  2025  and 
addressing  security  issues.  The  scenarios  were  called 
“Democracy  for  all”,  “Renaissance  for  realpolitik”, 
“Usama’s  triumph”  and  “Baltic  focus”. 

The  scenarios  were  used  to  show  that  some  ground 
fighting  capability  would  be  required  in  all  scenarios,  but 
also  that  the  conditions  and  requirements  regarding  that 
force  would  be  very  different  in  the  four  futures. 
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useful  option  for  this  type  of  uncertainty,  but  also  for  a  situation  where  dynamic  and  stochastic  uncertainty 
exists. 

A  rather  different  approach  to  addressing  uncertainty  is  Assumption-Based  Planning  [15].  It  is  not  limited 
to  any  particular  type  of  uncertainty,  but  is  used  to  revise  and  strengthen  an  already  accepted  plan. 
In  essence,  it  is  designed  to  identify  critical  assumptions  in  order  to  suggest  shaping  and  hedging  actions 
that  will  improve  an  existing  plan. 

The  fundamental  point  about  uncertainty  is  that  it  has  to  be  managed  properly  in  the  study  process, 
and  while  it  cannot  be  eliminated  its  implications  can  be  understood.  Importantly,  the  analyst  should  not 
treat  it  as  a  commodity  to  hand  on  to  the  decision  maker. 


6.4  STAKEHOLDER  ANALYSIS 

At  a  (very)  early  stage  of  the  study,  the  analyst,  in  partnership  with  the  primary  client  and  possibly  other 
select  members  of  the  client  system  (ref.  Chapter  4),  normally  conducts  a  stakeholder  analysis  exercise 
([16],  [17]).  The  objective  is  to: 

•  Develop  a  list  of  stakeholders; 

•  Identify  the  stakeholders’  key  characteristics; 

•  Identify  potential  risks  to  the  study  (e.g.  uncooperative  stakeholders  and  their  possible  adverse 
effects  on  the  study);  and 

•  Develop  insight  into  how  stakeholders  should  be  engaged  and  managed  during  the  study  (e.g.  how 
much  attention  to  devote  to  each  stakeholder,  which  stakeholders  to  invite  to  workshops,  which 
stakeholders  to  include  in  formal  communications  regarding  study  outputs,  how  to  manage 
pluralistic  competing  stakeholder  demands). 

The  first  of  the  following  two  sub-sections  addresses  what  aspects  of  stakeholders  should  be  analysed  and 
what  the  challenges  to  this  analysis  are.  The  second  sub-section  will  briefly  discuss  some  examples  of  how 
an  analysis  can  be  conducted  in  the  form  of  stakeholder  mapping. 

6.4.1  The  Challenges  of  Analysing  Stakeholders 

A  stakeholder  analysis  is  conducted  as  part  of  the  (initial)  problem  framing  process.  It  is  repeated  regularly 
during  the  study  on  a  (preferably)  proactive  or  on  a  reactive  basis.  Revisiting  and  revising  is  required  since 
the  stakeholder  community  is  not  static  (e.g.  as  the  study  progresses  through  its  lifecycle  the  members 
within  the  community  may  change  and/or  the  perceptions  of  individuals  may  change).  The  stakeholder 
analysis  may  need  to  be  kept  private  as  the  primary  client  may  be  uncomfortable  with  an  explicit  record 
(see  also  Chapter  3). 

For  each  stakeholder  who  is  identified,  assessments  may  be  made  of  the  following: 

•  The  stake  in  the  study; 

•  The  awareness  of  the  study; 

•  The  interest  in  the  study  (e.g.  why  is  he  interested,  what  are  his  expectations  and  objectives,  is  he 
directly  or  indirectly  affected); 

•  The  perception  regarding  the  study  (e.g.  does  he  believe  that  he  will  gain  an  advantage  or  be 
disadvantaged  through  the  study); 
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•  The  position  (e.g.  internal  or  external  actor,  organisational  membership,  title  or  rank,  place  in  the 
organisational  hierarchy); 

•  The  power  and  influence  (e.g.  ability  to  support  or  to  sabotage  the  study  process  and  resulting 
actions,  ability  to  influence  based  on  his  network  of  relationships,  his  importance  within  his 
organisation,  his  resources); 

•  The  relationships  with  other  stakeholders  (e.g.  potential  for  conflict,  potential  to  act  as  an  opinion 
leader,  which  stakeholders  have  similar  interests  and  views,  possibilities  for  forming  coalitions); 

•  The  attitude  towards  the  study  (e.g.  supportive,  neutral,  obstructive); 

•  The  legitimacy  in  relation  to  the  organisation; 

•  The  urgency  of  the  claim  on  the  organisation  (e.g.  how  important  is  the  study  to  the  stakeholder 
and  how  prepared  is  he  to  act  (supportively  or  obstructively)  to  achieve  his  own  desired  outcomes); 
and 

•  The  knowledge,  experience  and  expertise  in  areas  of  interest  to  the  study  (e.g.  how  could  he 
contribute  to  the  study,  participation  possibilities). 

A  stakeholder  analysis  is  by  nature  subjective  and  incomplete: 

•  There  are  uncertainties  and  inconsistencies  in  participant  assessments; 

•  There  is  always  a  residual  uncertainty  about  the  will  and  purpose  of  the  stakeholders  (even 
involved  stakeholders  may  choose  not  to  disclose  all  their  intentions); 

•  The  information  on  external  stakeholders  in  particular  will  be  difficult  to  determine  and  will  be 
incomplete  (e.g.  biased  assumptions,  use  of  stereotypes);  and 

•  The  stakeholder  community  is  dynamic  (e.g.  members  influence  one  another  and  change  their 
stances,  coalitions  form  and  break  up,  new  members  enter  and  old  ones  leave,  power  shifts,  interests 
change). 

Addressing  stakeholders  within  the  study  team 
is  a  matter  of  conducting  effective  facilitation 
that  is  trusted  by  the  stakeholders.  If  the  group 
of  stakeholders  that  needs  to  be  involved  in  a 
certain  study  phase  is  too  large  for  conducting 
a  single  workshop,  then  a  series  of  workshops 
may  be  conducted  or  other  approaches  may  be 
found  to  engage  all  key  actors  (e.g.  workshops 
supplemented  by  interviews).  The  benefit  of 
involving  stakeholders  is  that  they  may  voice  their  concerns  and  find  common  ground  in  a  problem 
definition,  and  ultimately  be  supportive  of  actions  suggested  by  the  study.  Involving  a  larger  number  of 
stakeholders  may  also  increase  the  quality  of  the  problem  definition,  as  each  stakeholder  brings  in  a  new 
perspective  on  the  issue.  The  drawback  may  be  that  stakeholders  with  sufficiently  divergent  interests  may 
block  any  development  of  a  common  view  and  may  get  information  that  may  benefit  a  contradictory 
purpose.  Some  may  even  attempt  to  bend  the  study  to  suit  their  own  ends. 

6.4.2  Stakeholder  Mapping 

Decisions  have  to  be  made  on  how  to  address  the  stakeholders  within  the  study  context.  Using  the  analysis 
discussed  in  Section  6.4.1,  a  stakeholder  mapping  can  be  designed  to  help  categorise  stakeholders  and 
identify  ways  of  coping  with  them.  Stakeholders  with  high  power  and  high  interest,  for  example,  are  likely  to 
be  critical  to  the  implementation  of  the  study’s  recommendations.  Some  of  those  stakeholders  may  perceive 
themselves  to  be  winners,  and  others  may  perceive  themselves  to  be  losers,  within  the  study  context. 


A  seminar  war  game  (SWG)  on  the  Canadian  Army  of 
Tomorrow  examined  military  operations  of  the  future  by 
incorporating  both  military  and  civilian  (e.g.  police, 
diplomats,  aid  agencies,  scientists,  and  others)  judgements 
and  perspectives.  It  provided  a  means  to  ‘meld’  these 
diverse  judgements  together  in  order  to  produce  insights 
into  future  operational  challenges  and  opportunities. 
‘Soft’  OA  helps  engaging  stakeholders  who  come  from 
different  cultures  and  have  very  different  points  of  view. 
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The  stakeholder  mapping  provides  a  visual  representation  of  some  of  the  key  parameters  useful  in 
characterising  the  stakeholders.  Depending  on  the  technique  used,  the  mapping  may  be  in  tabular,  graphical 
or  pictorial  format  (see  for  example  [16],  [17]  for  a  more  in-depth  analysis). 

For  example,  a  grid  may  be  drawn  with  power  (or  influence;  i.e.  the  capability  to  affect  the  study  and 
achieve  outcomes)  along  one  axis  and  interest  (in  the  study  and  its  outcomes)  along  the  other.  The  scale 
for  power  may,  for  example,  be  chosen  to  correspond  to  a  high-low  categorisation.  The  scale  for  interest 
may,  for  example,  be  chosen  to  be  characterised  as  high-low  or  as  similar-divergent.  Stakeholders  can  then 
be  placed  within  the  quadrants  based  on  their  power  and  interest. 

Other  examples  would  include:  opposing  power  to  dynamism  (predictability  of  a  stakeholder’s  stance), 
or  power  to  legitimacy  (of  the  actions  of  the  stakeholder  in  terms  of  desirability  or  appropriateness)  and  to 
urgency  (the  study’s  criticality  and  time-sensitivity  as  perceived  by  the  stakeholder). 

The  clustering  of  the  stakeholders  provides  an  indication  of  possible  stakeholder  groupings  in  terms  of  their 
degree  of  importance  to  the  study’s  success,  and  possible  management  strategies  for  each  grouping. 
Examples  of  management  strategies  for  each  of  the  quadrants  or  combinations  include,  but  are  not  limited  to: 

•  Keep  the  stakeholder  informed; 

•  Have  stakeholders  actively  participate; 

•  Try  to  influence  stakeholder;  and 

•  Monitor  stakeholders. 

The  analyst  conducting  the  stakeholder  mapping  needs  to  consider  his  own  roles,  objectives  and 
relationships  with  the  stakeholders.  Successful  stakeholder  negotiation  needs  to  give  due  consideration  to 
both  voice  (i.e.  participation)  and  procedural  justice  (i.e.  agreement  on  the  fairness  of  the  rules  followed 
during  collaboration). 


6.5  CHOOSING  THE  METHOD 

As  far  as  the  study  methodology  is  concerned,  there  is  a  decision  to  be  made  if  the  study  will  use  an 
approach  based  on  a  single  method  or  methodology  or  a  combination.  The  next  two  sections  will  address 
each  type  of  approach  separately.  This  section  and  its  sub-sections  will  address  the  modelling  process  in 
general  terms  and  those  problematic  situations  where  the  study  approach  in  fact  relies  on  one  single 
methodology  or  method.  This  will  most  likely  be  the  case  with  problems  having  a  narrow  scope,  a  limited 
aim,  or  with  separate  (partial)  studies  each  addressing  a  particular  stage  or  activity  of  a  larger-scale  study. 
Practical  experience  suggests  that  multi-methodology  (to  be  treated  in  Section  6.6)  will  be  the  more  usual 
choice  when  facing  messy  problematic  situations  and  certain  more  ‘messy’  instances  of  problems  proper. 

6.5.1  Deciding  on  the  ‘Soft’  OA  Method 

The  following  criteria  are  suggested  for  consideration  when  deciding  on  which  methodology/method  is 
appropriate  for  addressing  the  problematic  situation  at  hand  (partly  based  on  [18],  [19],  [20]): 

•  Fitness  for  purpose  of  the  task  to  be  conducted  and  the  type  of  results  desired.  This  includes  the 
following  aspects:  the  nature  of  anticipated  outcomes,  conformity  with  the  divergent  or  convergent 
stage  of  the  study3,  mental  demands  made  of  clients,  but,  most  important  and  in  fact  integrating  the 
previous  aspects,  appropriateness  for  developing  and/or  using  the  model  to  represent  the  problematic 
situation  or  support  dealing  with  it. 


3  Particularly  where  group  work  is  involved,  there  may  be  a  need  for  facilitating  both  divergent  and  convergent  thinking. 
As  both  types  of  thinking  usually  require  different  supporting  methods,  the  notion  of  mixing  methods  (‘multi-methodology’) 
becomes  natural  (ref.  Section  6.6  and  [20]). 
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In  each  phase  of  the  study  (ref.  Chapter  5)  decisions  have  to  be  made  concerning  the  general 
method  and  specific  techniques  to  use  for  the  different  tasks  in  that  phase;  each  of  them  should  be 
fit  for  their  purpose. 

•  The  nature  and  degree  of  uncertainty 
involved  and  to  what  extent  it  should 
be  coped  with  by  the  method. 

This  criterion  extends  the  previous  one 
if  uncertainty  is  a  fundamental 
characteristic  of  the  problematic 
situation  the  methodology/method  is 
designed  for. 

•  The  degree  to  which  a  candidate 
method  is  accepted  as  formally  sound 
or  grounded  in  theory  and/or  proven  to 
be  successful  in  similar  circumstances. 

•  Pragmatic  reasons  based  on,  among  others,  time,  budget  and  availability  of  people  to  be  involved. 

•  Data  requirements  and  other  model-related  resources  such  as  availability  of  software  and  other 
supporting  material. 

•  The  analyst ’s  knowledge  of,  competence  in  and  experience  with  applying  the  (mechanics  of  the) 
method. 

•  The  facilitator’s  knowledge  of,  competence  in  and  experience  with  ways  (‘methods’)  in  which 
groups  should  be  guided  and  stimulated  to  achieve  results,  including  coping  with  issues  of 
dysfunction  and  disagreement,  if  the  method  requires  group  sessions.  If  not,  are  additional 
facilitators  available  and  able  to  fit  in  the  team? 

•  The  degree  to  which  the  clients  and  other  stakeholders  understand  and  feel  comfortable  with  the 
proposed  method. 

•  The  degree  to  which  the  organisational  culture  agrees  with  the  proposed  method. 

Below  follows  an  example  showing  how  the  nature  of  a  problematic  situation  influences  the  nature  of  the 
methodology  to  be  chosen. 


One  of  the  factors  considered  in  the  choice  of  an  analytical 
approach  to  examine  issues  related  to  the  Canadian  Army 
of  Tomorrow  was  the  cost  of  the  study.  Conducting  a 
seminar  war-game  (SWG)  required  less  resources  and  less 
time  than  simulation-supported  experiments  and  field  trials 
with  military  personnel.  However,  the  SWG  was  able  to 
contribute  valuable  insights  for  filtering  out  less  promising 
alternatives,  for  structuring  future  studies  and  for 
determining  who  from  the  diverse  stakeholder  systems 
should  have  a  role  in  those  studies. 

The  SWG,  as  an  example  of  ‘soft’  OA,  is  sometimes  (but 
not  always)  a  less  resource  intensive  alternative. 


Table  6-1:  Relationship  Between  Problem  Nature  and  Methodology  (Adapted  from  [21]). 


Nature  of  Problematic  Situation 

Nature  of  Methodology 

Very  complex  and  important  to  stakeholders. 

Should  increase  quality  of  communication. 

Very  political  where  stakeholders  have  hidden 
agendas. 

Should  visualise  and  relate  issues,  impartially 
facilitated. 

Very  data  rich  where  stakeholders  aim  for  concrete 
options. 

Should  integrate  opinions  and  data  and  create 
options  and  show  consequences. 

Creating  a  model  which  is  a  representation  of  the  problematic  part  of  (real  or  perceived)  reality  that  people 
wish  to  deal  with,  requires  a  methodology  or  a  method  in  itself.  This  (procedural)  methodology  or  method 
is  a  structured  set  of  guidelines  or  rules  and  will  be  applied  by  the  analyst  or  by  a  group  of  people  (in  a 
‘facilitated  modelling’  mode).  The  application  of  different  (procedural)  methodologies/methods  for  setting 
up  a  model  will  most  likely  produce  different  models  for  analysis.  None  of  them  may  be  correct,  but  some 
of  them  will  be  more  useful  than  others. 
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Some  of  the  methods  in  the  field  of  judgement-based  OA  are  strongly  associated  with  individual  academic 
researchers  and  practitioners,  often  those  who  first  characterised  the  method,  exploited  it  and  published 
their  experiences  of  using  it.  Some  methods  are  asserted  to  be  exclusively  useful  for  specific  types  of 
problematic  situations,  e.g.  the  exploration  of  strategic  issues  for  organisations.  Practitioners  wishing  to 
apply  a  specific  method  are  advised  to  take  an  open  approach  and  examine  all  methods  on  their  merits  in 
the  problematic  situation  on  hand  before  deciding  which  method  to  use.  This  state  of  affairs  perhaps 
reflects  the  maturity  of  ‘soft’  OA.  In  a  more  established  science,  such  as  physics,  there  are  standard 
approaches  to  tackling  issues.  This  convergence  has  not  yet  emerged  in  ‘soft’  OA  and  while  there  are 
standard  methods  such  as  influence  diagrams,  there  is  still  individual  preference  in  choice  of  methodology. 

The  reasons  why  a  particular  method  is  chosen  should  be  documented  and  accounted  for.  The  same  holds 
when  it  has  become  apparent  that  it  is  necessary  to  deviate  from  the  formalities  of  a  particular  method  or 
to  adopt  a  multi-methodology  approach. 


6.5.2  Deciding  on  the  Model’s  Resources 

There  are  two  related  issues  that  must  be  dealt  with  when  developing  a  model  and  applying  a  method: 
the  nature  and  the  origin  of  data.  On  the  one  hand,  these  two  determine  the  way  in  which  data  is  collected 
and  used  as  input  to  a  model.  On  the  other  hand,  they  depend  on  the  type  of  model  which,  in  turn,  depends 
on  the  study’s,  or  study  activity’s,  purpose  for  which  the  model  will  be  constructed  and  the  type  of  results 
that  are  desired. 

Figure  6-3  shows  how  a  series  of  requirements  originate  from  the  aim  of  the  study,  leading  to  data 
resource  requirements.  The  nature  and  origin  of  the  data,  in  combination  with  other  intrinsic  data  quality 
factors,  affect,  in  turn,  the  model  and  its  results  and  thus  the  achievement  of  the  study  aim.  Also  with  the 
model  and  its  results  there  are  additional,  more  intrinsic,  quality  factors  that  affect  the  series  of  achievements, 
as  shown.  In  the  end,  there  may  be  a  mismatch  between  what  the  study  aim  required  and  what  could  be 
achieved. 


iteration  to 
achieve 
requisiteness 
regarding 
model  /  data 

mismatch? 


intrinsic  quality: 
e.g.  reliability/ 
unbiasedness, 
precision, 
relevance 


intrinsic  quality: 
e.g.  correctness, 
supportiveness, 
richness, 
appropriateness 


intrinsic  quality: 
e.g.  robustness, 
concurrence, 
meaningfulness, 
relevance 


Figure  6-3:  Data  and  a  Possible  Mismatch  Between  Requirements  and  Achievements, 
in  Addition  to  Other  Intrinsic  Quality  Factors  Per  Key  Study  Component 
(This  may  be  an  iterative  process  until  requisiteness  is  achieved;  ref.  Section  5.4). 


Figure  6-3  is  valid  for  OA  in  general,  but  in  ‘soft’  OA  data  largely  depend  on  people  and  their  judgements 
and  much  less  on  numbers  and  figures.  This  adds  a  particular  dimension  to  the  subject  as  these  judgements 
represent  people’s  beliefs  and  assumptions  about  the  problematic  situation,  their  preferences,  and  their 
estimates  of  otherwise  unavailable  numerical  data.  Chapter  2’s  Figure  2-3  makes  a  distinction  between 
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quantitative  and  qualitative  data,  and  between  objective  and  subjective  data.  The  nature  of  data  can  be 
manifold;  commonly  accepted  theory  however  defines  a  limited  number  of  formal  scales  on  which  data 
can  be  expressed  and  phenomena  can  be  measured. 


6.6  CHOOSING  A  COMBINATION  OF  METHODS  -  MULTI-METHODOLOGY 

Experience  has  shown  that  a  large  number  of  problematic  situations  require  combining  different 
methodologies  and/or  methods. 

While  acknowledging  the  already  multi-disciplinary  nature  of  OA  itself,  any  operational  analyst  should 
also  be  aware  of  possibly  appropriate  methods  and  methodologies  from  other  fields  of  analysis: 
economics,  psychology,  sociology,  political  sciences,  ethics,  etc.  This  requires  an  open  mind  and  out-of- 
the-box  thinking  by  the  operational  analyst  and  an  preparedness  to  call  in  or  at  least  consult  colleagues 
from  those  other  fields. 

Typically,  as  knowledge  in  a  problem  space  is  elicited  and  captured,  the  preferred  methods  become  more 
focussed  and  specific,  often  creating  opportunities  for  quantification  of  the  problem  space,  with  attendant 
statistical  analysis. 

Particular  care  needs  to  be  taken  where  ‘soft’  and  ‘hard’  methods  are  interleaved.  In  general,  the  relative 
simplicity  of  representation  achievable  by  a  ‘soft’  method  can  significantly  degrade  the  credibility  and 
resolution  of  predecessor  ‘hard’  techniques.  A  ‘hard’  method  can  successfully  address  the  interaction  of 
distinct  factors;  if  a  ‘soft’  method  is  subsequently  applied  to  the  outputs  of  that  ‘hard’  method,  there  is  a 
danger  that  the  impact  of  such  interactions  will  be  dissipated  by  the  architecture  of  the  ‘soft’  method. 
A  final  opinion-based  step  may  be  defensible  especially  if  it  involves  the  decision  makers  themselves. 
Difficulties  arise  in  particular  where  a  judgement-based  step  is  planned  between  the  application  of  two 
‘hard’  methods. 

Noting  the  caveat  above,  even  when  relying  on 
‘hard’  OA  methods,  ‘soft’  approaches  may  be 
helpful  for  the  initial  problem  shaping  and 
structuring  tasks  of  the  Appreciation  and 
Assessment  phases;  perhaps  for  its  subsequent 
Analysis  and  Action  phases  as  well,  in  which, 
for  example,  a  range  of  measures  of  effectiveness 
need  to  be  embraced  and  interpreted  together. 

The  overriding  requirement  is  to  strive  to 
maximise  objectivity  and  rigour,  gauging  the 
validity  of  the  overall  approach  in  terms 
appropriate  to  the  problem  domain  being 
addressed. 

‘Multi-methodology’  is  a  rather  ambiguous  term 
as  it  can  be  understood  as: 

•  A  ‘ multi-method  (or:  ‘mixed-method’)  methodology’ :  a  combination  of  different  methods  from  a 
single  methodology  (e.g.  use  of  various  aggregation  methods  all  belonging  to  the  methodology  of 
value-based  multiple  criteria  decision  making);  and 

•  A  ‘ multi-method  multi-methodology’ :  a  combination  of  different  methods  from  different 
methodologies  (e.g.  use  of  a  mapping  method  belonging  to  the  methodology  of  conceptual  mapping 
and  a  value-based  multiple  criteria  decision-making  method).  One  of  the  methodologies  may  be 
dominant  and  will  then  be  enhanced  with  suitable  methods  from  one  or  more  other  methodologies. 


In  supporting  the  development  of  new  operational  concepts 

for  maritime  mine  counter  measures  a  multi-methodology 
approach  was  designed: 

•  Methods  used  in  the  1st  work  package  were: 
scenario  development  and  analysis,  capability  analysis. 

•  Methods  used  in  the  2nd  parallel  work  package  were: 
technology  survey  and  assessment. 

•  Methods  used  in  the  3rd  subsequent  work  package  were: 
morphological  analysis  (for  concept  design),  multi-criteria 
analysis  (for  concept  evaluation). 

•  Methods  used  in  the  4th  subsequent  work  package  were: 
additional  technological  analysis,  in-depth  multi-criteria 
analysis. 

The  above  is  an  example  of  a  problematic  situation  requiring 
addressing  multiple  problem  aspects  and  therefore  applying  a 
multitude  of  appropriate  methods. 
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The  CoBP  continues  using  the  term  ‘multi-methodology’  for  both  interpretations  as  has  become 
customary  in  the  ‘soft’  OA  literature4.  Some  authors  also  account  for  the  paradigm  level  (single  versus 
multi-paradigm;  ref.  Section  2.4)  when  defining  multi-methodology.  The  CoBP  will  not  address  that  issue 
as  it  takes  the  stance  that  ‘hard’  and  ‘soft’  OA  are  not  really  different  paradigms. 

6.6.1  Deciding  on  a  Multi-Methodology  Approach 

There  are  three  main  reasons  for  deciding  to  adopt  a  multi-methodology  approach  ([22],  [23]),  which  is  a 
common  and  successful  practice  among  experienced  and  knowledgeable  practising  analysts: 

•  The  problematic  situation  is  multi-dimensional  (ref.  Chapter  2):  there  are  multiple  problem 
aspects  and/or  distinguishable  problem  parts,  multiple  actors  with  multiple  perspectives  and 
multiple  objectives,  multiple  types  of  information  resources  and  sources  of  uncertainty,  which 
may  all  require  their  own  way  of  being  coped  with. 

•  The  study  process  is  multi-phased :  each  phase  may  have  a  different  nature  posing  different  tasks 
and  activities  and  thus  requiring  its  own  dedicated  methodology. 

•  The  usefulness  of  enhancement  and  triangulation:  different  methods  or  even  methodologies 
applied  to  the  same  issue  may  produce  different  results  which  can  widen  the  scope  of  the  study 
and  open  up  the  option  space  but  can  also  be  used  for  validation  and  for  coping  with  weaknesses 
of  particular  methods.  A  particular  example  would  be  that  requirements  of  one  method  uncover 
gaps  in  the  analysis  conducted  with  another  method. 

Sometimes,  using  different  methods  just  happens  as  the  study  evolves  and  insights  change. 

It  must  further  be  noted  that  a  multi-methodology  approach  can  be  designed  as  a  serial  or  as  a  parallel 
process,  or  even  a  combination.  A  serial  multi-methodology  approach  means  that  the  combination  of 
different  methods  and  methodologies  occurs  across  the  different  study  phases  or  even  across  subsequent 
activities  within  a  phase.  Each  subsequent  phase  or  activity  relies  on  a  specific  method  or  methodology. 
A  parallel  multi-methodology’  approach  (although  less  common)  means  that  different  methods  or 
methodologies  are  used  in  parallel  within  the  same  study  phase  (mainly  for  the  purpose  addressed  by  the 
third  bullet  above). 

Taking  the  above-mentioned  argument  of 
usefulness  further,  a  problematic  situation 
which  possesses  more  characteristics  of  a  mess 
or  even  a  problem  proper  than  of  a  puzzle,  will 
also  be  more  appropriate  for  the  use  of 
different  methods.  Using  different  modelling 
methods  will  most  likely  produce  different 
models  thereby  opening  up  different 
perspectives  of  the  situation  and  stimulating 
discussion;  this  is  an  example  of  parallel  multi¬ 
methodology.  Section  5.4  also  discusses  serial 
and  parallel  approaches. 

The  diagram  in  Chapter  3  (Figure  3-1)  shows  that  a  problem  structuring  analysis  may  identify  specific 
issues  that  need  exploiting  by  methods  from  ‘hard’  OA.  This  would  be  a  perfect  example  of  ‘soft’  and 
‘hard’  OA  complementing  each  other. 


AUK  study  of  the  value  for  money  of  the  Falcon 
communications  system  demonstrates  the  power  of 
complementarity.  The  system  has  the  potential  to  generate 
several  benefits,  each  assessed  by  a  different  combination 
of  techniques:  military  organisation  flexibility,  through  a 
network  model  informed  by  a  multi-criteria  analysis; 
enhanced  powers  of  manoeuvre,  by  logical  analysis  of 
interoperabilities;  operational  pace  and  advantage  by  a 
network  efficiency  model  based  on  judgement;  ‘operational 
picture’  completeness  through  a  ‘hard’  network  simulation 
of  data  flows;  and  battle  outcome  from  a  simulation  which 
drew  on  these  parameters. 


4  In  addition,  it  should  be  noted  that  multi-methodology  need  not  involve  mixing  whole  methods.  Elements  of  different  methods 
may  be  taken  together  to,  in  fact,  create  an  integrated  new  method.  An  existing  method  could  be  enhanced  (enriched)  by 
elements  of  another  method. 
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There  are  also  reasons  for  perhaps  not  adopting  a  multi-methodology  approach  if  it  is  felt  that  the 
challenges  listed  below  cannot  be  coped  with  ([1],  [22],  [24]): 

•  Some  schools  of  thought  (at  companies,  advisory  or  academic  institutions)  advocate  their  own 
preferred  methodology  and  will  not  be  prepared  to  use  another  methodology.  If  they  are  part  of  the 
same  study  team,  this  may  pose  a  cultural  and  organisational  barrier  to  a  multi-methodology 
approach.  This  has  to  be  identified  and  negotiated  at  as  early  a  stage  as  possible. 

•  Some  theorists  point  at  theoretical  difficulties  with  combining  certain  methodologies,  especially 
when  they  feel  that  methodologies  belong  to  different  paradigms.  In  practice  this  means  that  it 
may  not  be  clear  how  results  from  one  method  could  or  should  be  used  as  input  to  a  subsequent 
method.  Some  adjustment  or  inteipretation  may  be  necessary,  which  has  to  be  recorded. 
In  addition,  that  same  practice  shows  that  this  theoretical  issue  is  of  minor  importance  with  the 
exception  of  the  issue  of  interleaving  ‘hard’  and  ‘soft’  methods  (discussed  earlier  in  this  section). 

•  Along  similar  lines  as  the  previous  reason:  moving  from  one  methodology  to  another  may  pose 
cognitive  difficulties  to  analysts  as  this  movement  may  require  not  only  an  altogether  different 
analytical  attitude  but  also  specific  practical  skills.  There  may  exist  cognitive  difficulties  for 
clients  as  well  who  have  to  adapt  themselves  to,  for  example,  different  representations  of  problem 
issues.  The  analyst  therefore  needs  not  only  to  explain  what  is  expected  of  clients  but  also  to  seek 
assistance  if  his  expertise/knowledge  seems  insufficient. 

6.6.2  Designing  a  Multi-Methodology  Approach 

Recall  from  Chapter  5  the  four  main  activity- oriented  phases  of  a  study:  appreciation,  analysis,  assessment 
and  action.  They  can  also  be  used  to  design  a  multi-methodology  approach  assuming  that  each  phase’s 
nature  and  tasks  call  for  its  own  dominant  methodology  (or  method). 

Add  to  this  set  of  phases  three  ‘worlds’  in  order  to  create  a  full  matrix  of  different  dimensions  which  might 
act  as  a  framework  to  design  and  appreciate  the  elements  of  a  multi-methodology  approach.  These  ‘worlds’ 
(material,  social,  personal)  are  based  on  the  work  by  Habermas  [25],  but  will  for  the  purpose  of  the  CoBP  be 
loosely  interpreted  as  the  following  respective  domains'. 

•  The  objective,  observable  domain  (‘what  we  see’); 

•  The  inter-subjective,  participatory  domain  (‘what  goes  on  in  groups  we  participate  in’);  and 

•  The  subjective,  individual  domain  (‘what  we  think  and  experience  ourselves’). 

Table  6-2  shows  the  matrix-like  framework,  very  similar  to  the  one  Mingers  suggested  [1 9]5.  Each  cell 
contains  a  few  phrases  characterising  the  specific  combination  of  phase  and  domain  and  suggesting  the 
nature  of  the  methodology(ies)  to  consider.  The  actual  choice  does,  of  course,  depend  on  the  key 
ingredients  of  any  study  design  (ref.  Chapter  5):  the  problematic  situation  itself,  the  people  involved, 
the  nature  of  the  decision  to  make,  the  nature  of  the  data  and  information  required  to  produce  outcomes 
that  can  inform  the  decision. 


5  An  alternative  approach  for  choosing  a  combination  of  methodologies  is  ‘Total  Systems  Intervention’  developed  by  Flood  and 
Jackson  [27].  Mingers’  approach  fits  more  closely  with  the  terminology  and  study  phasing  adopted  in  this  CoBP. 
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Table  6-2:  Framework  for  Designing  a  Multi-Methodology  Approach  Using 
the  General  Study  Phasing  of  Figure  5-2  (Adapted  from  [19]  and  [22]). 


Study  Phases-> 
Domains  d/ 

Appreciation 

of: 

Analysis 

of: 

Assessment 

of: 

Action 
to  be  Taken: 

Objective/Observable 

(Physical) 

circumstances; 

organisational 

demands. 

Correlations  and 
causal  structures; 
external 
influences. 

Alternative 
options  and  their 
impacts. 

Select  and 
implement. 

Intersubjective/- 

Participatory 

Power  relations; 
group  dynamics. 

Conflicts; 

common 

interests. 

Alternative 

(social) 

structures. 

Generate 
consensus  and 
mutual 

empowerment. 

Subjective/Individual 

Emotions;  beliefs; 
meanings. 

Perspectives; 

perceptions; 

rationality. 

Alternative 

conceptualis¬ 

ations. 

Generate 
agreement  and 
commitment. 

The  following  steps  will  usually  have  to  be  followed  for  making  that  choice  [23]: 

•  Review  the  problematic  situation  and  the  state  and  progress  of  the  study; 

•  Determine  what  is  required  to  make  a  further  step  toward  the  next  study  phase  (and  thus  toward 
the  aim  of  the  study); 

•  Be  aware  of  the  methods  and  methodologies  that  are  available  for  what  is  required  and  understand 
which  of  them  are  useful  and  in  what  ways;  and 

•  Choose  the  most  appropriate  methods  taking  into  account  not  only  criteria  related  to  the  issues 
mentioned  in  the  previous  steps  and  to  identified  uncertainty  types,  but  also  to  study  constraints 
(budget,  time,  agenda,  etc.)  and  analytical  or  other  requirements  (skills  and  experience,  cognitive 
demands,  cultural  barriers,  etc.). 

Attempting  to  position  candidate  methods  in  the  cells  of  this  framework  will  not  only  show  which  study 
aspects  can  be  covered  but  also  which  cannot  and  thus  remain  open  (the  reader  is  referred  to  [26]  for 
examples).  A  discussion  should  be  organised  between  analyst  and  client  about  what  to  do  with  a  ‘white 
spot’  and  whether  or  not  it  might  be  acceptable  or  whether  a  new,  possibly  modified,  method  or  methodology 
should  be  designed. 

The  most  important  matter  of  attention  for  an  analyst  proposing  a  multi-methodology  approach  should  be 
to  explain  and  discuss  the  proposed  methodology  and  gain  commitment  with  the  primary  client  and 
preferably  with  other  involved  key  individuals  as  well. 

From  practice  [1]  it  must  be  concluded  that  the  choice  of  methods  evolves  during  a  study  based  on 
reflections  along  the  way  and  can  therefore  not  be  fixed  at  the  study’s  beginning.  In  addition,  the  choice  of 
the  methods  and  methodologies  to  be  combined  largely  depends  on  the  knowledge,  experience  and 
competence  of  the  analyst  as  noted  earlier.  Any  analyst  must  consciously  reflect  on  his  methodological 
decisions  and  be  prepared  to  consult  with  colleagues;  he  should  even  ask  them  to  join  the  team  if  and  when 
the  success  of  a  study  (and  the  satisfaction  of  the  client)  does  so  require. 

6.6.3  Applying  a  Multi-Methodology  Approach 

Applying  a  multi-methodology  approach  needs  continuous  reflection  on  at  least  the  following  four  issues 
(inspired  by  [20],  [28],  [29]): 
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•  The  division  of  roles  between  the  analyst(s)  and  the  facilitator(s),  and  among  analysts  and  among 
facilitators  if  there  are  more  than  one. 

Refer  to  Chapter  4  for  remarks  on  the  differences  between  an  analyst  and  a  facilitator.  Each  has  his 
own  role  to  play,  in  principle,  but  the  roles  can  be  played  by  one  single  individual.  Although  there 
can  be  one  principal  analyst  or  facilitator  who  is  most  familiar  with  a  particular  method  used  within 
the  study,  it  is  wise  to  have  basic  knowledge  of  that  method  present  with  all  analysts/facilitators. 
Knowledge  overlap  will  contribute  to  informed  assessment  of  the  suitability  of  the  elements  of  a 
multi-methodology  approach,  the  successful  integration  of  the  results,  and  help  ensure  that  informed 
support  (e.g.  technical  support,  use  of  software)  can  be  provided  by  the  non-principal  analyst/ 
facilitator. 

•  The  translation  of  results  from  one  method  to  another. 

The  results  of  one  method  may  not  be  in  the  correct  format  for  or  may  be  otherwise  unsuitable  as 
input  to  another  method.  For  example,  a  causal  map  structure  containing  cycles  of  influence 
cannot  be  used  as  such  as  a  formal  value-based  multi-criteria  model  which  usually  cannot  cope 
with  cycles.  So,  there  has  to  be  a  translation  of  one  to  the  other.  This  is  a  non-obvious  task  to  be 
carried  out  by  skilful  analysts,  but  surprisingly  little  reported  on  in  the  literature. 

•  The  combination  of  results  from  different  methods  to  the  overall  study  outcome. 

Although  the  initial  design  may  be  a  serial  approach  (each  subsequent  study  phase  requiring  its  own 
single  method  contributing  to  the  overall  study  aim),  it  can  happen  that  for  reasons  of  enrichment  or 
triangulation  or  challenging  or  opening  up  the  options  space  (ref.  Section  6.5.1),  a  parallel  approach 
is  adopted.  The  different  methods  applied  in  a  single  study  phase,  all  contributing  to  the  anticipated 
outcome  of  that  phase  may  pose  a  challenge  as  to  the  nature  of  the  combined  results. 

At  the  results  level  of  the  individual  methods  there  can  be  mutual  confirmation,  i.e.  results  all  point 
to  the  same  direction,  but  also  contradiction  (see  also  Figure  6-3).  In  the  latter  case,  there  has  to  be 
some  degree  of  reconciliation  at  the  outcome  level  (outcome  based  on  the  results)  in  order  to  allow 
study  progress.  This  can  be  achieved  by  a  discussion  of  the  pros  and  cons,  meanings  and 
consequences,  and  reliability  and  other  caveats  of  the  various  findings. 

•  The  validity  of  the  end-to-end  integrity  of  the  study. 

Moving  from  study  phase  to  study  phase  and  subsequently  applying  methods  may  lead  to  new 
insights  along  the  way,  but  also  to  the  danger  of  loosing  sight  of  the  over-arching  study  philosophy 
and  its  initial  assumptions  (e.g.  about  the  features  of  the  problematic  situation).  Adjustment  of  these 
defining  elements  according  to  the  new  insights,  for  example  by  updating  an  initial  cognitive  map, 
will  help  prevent  losing  the  study’s  end-to-end  integrity  and  provide  a  record  of  the  study’s  progress. 


6.7  COMPUTER  SUPPORT 

‘Soft’  OA  can  be  conducted  using  high,  medium  or  low  technology  approaches.  When  computers  are 
used,  they  may  be  used  interactively  during  sessions  with  the  participant(s)  (e.g.  interactively  building  and 
refining  models  visualising  and  structuring  the  problematic  situation)  or  after  such  sessions  in  an  analysis 
mode.  A  specialised  use  of  computers  in  facilitated  workshops  (ref.  Chapter  7  on  workshop  facilitation) 
has  arisen  through  the  use  of  keyboard-entered,  rather  than  oral,  contributions.  Anonymous  contributions 
may  be  made,  allowing  speculative  thinking  and  avoiding  domination  by  senior  military  officers. 

Computers,  because  they  are  useful  for  storing,  retrieving,  manipulating  and  communicating  information, 
can  be  a  powerful  tool  that  helps  the  facilitator  and  the  group  to  accomplish  their  tasks  [30].  Computers 
may  also  be  used  in  modes  that  provide  anonymity  to  participants  and  so  help  manage  personality  issues 
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and  pressures  resulting  from  military  rank  differentials  among  participants.  However,  their  use  should  not 
be  allowed  to  intrude  too  much  on  group  interaction.  Participants  concentrating  too  much  on  computer 
screens  rather  than  on  each  other  may  neglect  building  the  group  relationships  and  dynamics  that  are 
critical  for  the  development  of  robust  solutions. 

At  all  stages  of  a  study  there  may  be  an  opportunity  for  or  even  a  need  for  computerised  support  of  the 
modelling  process6.  This  however  requires  considering  very  thoroughly  the  aspects  discussed  in  Sections 
6.5.1,  6.6.1  and  6.6.3  (see  also  [32]),  as  computerised  models  can  not  only  be  helpful  and  enhance  process 
efficiency  and  effectiveness  but  can  also  be  constraining  to  some  degree.  It  should  not  be  taken  for  granted 
that  some  model  software  perfectly  matches  the  requirements  of  a  problematic  situation.  In  addition,  in  a 
multi-methodology  approach,  usually  some  effort  is  required  to  transfer  output  from  one  computer  model  to 
input  for  another.  Moreover,  some  technical  knowledge  may  be  required  to  operate  the  software;  a  technical 
facilitator  will  usually  be  needed  for  support.  Nevertheless,  problem  structuring,  visualising  models, 
executing  calculations  for  those  parts  of  the  analysis  that  require  some  ‘hard’  OA  techniques,  comparing 
options,  investigating  their  (possibly  time-dynamic)  consequences  and  synthesising  information,  visualising 
results,  all  may  and  indeed  have  proven  to  benefit  from  computer  support  (e.g.  see  [10]  for  computer- 
supported  problem  structuring  and  scenario  creation  using  Morphological  Analysis).  The  level  of  computer 
support  may  vary  considerably  though;  some  ‘soft’  approaches  rely  almost  exclusively  on  it  [32], 
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•  Chapter  1  -  INTRODUCTION 

•  Chapter  2  -  PROBLEMATIC  SITUATIONS  AND  'SOFT'  OA 

•  Chapter  3  ■  ACHIEVING  VALIDITY,  CREDIBILITY  AND  ACCEPTANCE 


•  Chapter  4-  ROLES  AND  RESPONSIBILITIES 


•  Chapter  5  -  THE  'SOFT'  OA  STUDY  PROCESS 

•  Chapter  6  -  DESIGNING  THE 'SOFT'  OA  STUDY  METHODOLOGY 

•  Chapter  7-  DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 

•  Chapter  8  -  INTERPRETING,  USING  AND  COMMUNICATING  RESULTS  AND  OUTCOMES 


•  Chapter  9  ■  THE  ESSENCE  OF  THE  CODE  OF  BEST  PRACTICE 

•  Annex  A  -  LIST  OF  KEY  STATEMENTS  AND  RECOMMENDATIONS  OF  IMPORTANCE 

TO  THE  ANALYST 

•  Annex  B  ■  SELECTIVE  LIST  OF  'SOFT'  METHODS 


>  Understand  what  the  three  viewpoints  for  using  humans  as  data  sources  mean  for  the  study:  the 
cognitive,  critical  and  constructive  viewpoints.  Understand  types  of  bias  that  are  recognised  within 
each  perspective’s  context. 

>  Consider  carefully  what  stakeholders  (including  subject  matter  experts)  to  include  in  the  study,  and 
how  they  will  be  invited  to  participate. 

>  Design  facilitated  workshops  in  order  to  engage  with  stakeholders. 

>  Select  the  key  conditions  in  the  Schuman  model  that  are  considered  to  be  critical  to  conducting 
workshops  in  the  current  study. 

>  Ensure  that  the  ‘theory  of  action’  underpinning  any  considered  course  of  action  is  understood  and 
documented. 

>  Design  (additional)  data  gathering  to  be  conducted  outside  a  workshop  setting,  including  what 
sampling  strategy  is  used. 

>  Consider  using  the  Human  Environment  Analysis  Reasoning  Tool. 

>  Document  all  data  collection  measures  taken  in  a  study,  including  the  methods,  the  sources  and  the 
collectors. 


RTO-TR-SAS-087 


7-1 


DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 


ORGANIZATION 


7.1  INTRODUCTION 

This  chapter  discusses  approaches  for  collecting  and  analysing  judgement-based  data.  This  chapter  assumes 
that  data  collection  is  guided  by  four  general  premises  [1]: 

•  Premise  1 :  Problems  are  socially  constructed  entities. 

•  Premise  2:  Subjectivity  is  unavoidable. 

•  Premise  3:  Clients  want  ‘satisficing’  solutions. 

•  Premise  4:  Participation  increases  commitment  for  implementation. 

These  premises  will  infonn  how  data  is  viewed  and  the  choice  of  data  collection  methods.  This  is  discussed 
in  the  first  part  of  the  chapter.  The  remainder  of  the  chapter  then  discusses  using  workshops  for  data 
collection,  other  data  collection  approaches,  and  the  use  of  Subject-Matter  Experts  (SMEs). 

7.2  THE  PROBLEMATIC  NATURE  OF  DATA  IN  ‘SOFT’  OA 

Collection  of  input  material  to  help  with  formulating  and  characterising  problems  is  a  key  aspect  in  ‘soft’ 
OA.  Such  inputs  are  usually  referred  to  as  ‘data’  though  it  is  recognised  that  they  are  part  of  a  hierarchy  or 
pyramid  comprising  data,  information,  understanding,  knowledge  and  wisdom  [2],  We  will  not  try  to 
differentiate  the  lower-positioned  concepts  in  this  model  but  will  note  that  there  is  deeper  understanding  of 
system  behaviour  as  we  move  towards  wisdom.  Thus  we  will  use  data  in  its  broadest  sense  -  material 
(i.e.  pieces  of  information)  that  we  do  not  initially  know  but  is  potentially  required  in  the  study.  Importantly, 
data  should  not  be  considered  as  merely  quantitative  properties  of  well-defined  constituent  items.  In  addition, 
we  might  need  to  know  about  procedures,  interactions  of  system  components,  constraints  and  bounds, 
ownership  of  elements,  relative  importance  and  weightings,  etc.;  these  are  all  data  relevant  to  the  problematic 
situation. 

Different  views  on  using  humans  as  sources  of  data  exist,  originating  from  different  views  on  how  valid 
knowledge  is  created.  As  we  attempt  to  mix  methods  from  ‘hard’  and  ‘soft’  approaches,  none  of  them  can 
be  ignored  in  an  investigation  using  human  sources.  In  this  chapter,  these  views  will  be  called  the 
cognitive,  critical  and  constructive  viewpoints. 

7.2.1  The  Cognitive  Viewpoint 

In  this  viewpoint,  humans  are  seen  as  fallible  observers  of  reality.  Due  to  limitations  in  the  cognitive 
system  errors  or  biases  appear  -  different  errors  in  information  processing  may  appear  in  different  stages 
of  the  cognitive  system  [3],  such  as  the  following: 

•  The  ease  with  which  information  can  be  recalled  from  memory  affects  how  frequently  a  given 
type  of  event  is  deemed  to  be.  Well-publicised  events,  such  as  tornadoes,  may  be  assessed  to  be 
more  frequent  than  they  actually  are. 

•  The  mind  does  not  register  what  it  cannot  perceive.  We  cannot  know  the  actual  performance  of  an 
option  that  was  considered  but  subsequently  rejected.  In  a  game  of  chance,  there  is  thus  an  inclination 
to  believe  that  the  actual  outcome  was  more  likely  than  it  would  otherwise  have  been  deemed 
before  the  event. 

•  How  information  is  presented  affects  how  it  is  remembered.  The  first  and  last  element  in  a  list  is 
more  likely  to  be  remembered.  Easily  available  information  may  be  prioritised  over  information 
that  has  to  be  sought  after. 

•  Concrete  information  is  more  salient  than  abstract  information.  A  vivid  description  is  more  easily 
recalled  than  statistically  summarising  data.  One  personal  experience  of  a  failure  may  be  given 
more  weight  than  statistics  that  show  the  system  in  question  is  actually  quite  reliable. 
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•  What  a  person  expects  to  see  dramatically  affects  how  reality  is  perceived.  Playing  cards  with  red 
spades,  for  instance,  is  extraordinarily  difficult. 

•  There  is  also  a  confirmation  bias,  i.e.  people  tend  to  seek  information  that  is  consistent  with  their 
hypothesis  rather  than  information  that  could  cause  them  to  reject  a  hypothesis. 

•  A  special  case  occurs  when  experts  are  expected  to  quantify  their  input,  such  as  when  giving 
probabilities.  If  a  person  is  an  expert  in  a  specific  area,  his  judgement  may  be  vulnerable  to 
particular  biases  when  assigning  probabilities,  but  general  risk  management  experts  may  be  less 
affected  ([4],  [5]). 

•  A  well-known  bias  associated  with  workshops  is  group-think.  Symptoms  of  group-think  include: 

•  The  illusion  of  invulnerability  creating  excessive  risk-taking; 

•  Collective  efforts  to  discount  warnings  that  might  lead  members  to  reconsider  their  assumptions; 

•  An  unquestioned  belief  in  the  inherent  morality  of  the  group; 

•  Stereotyped  views  of  rivals  and  enemies;  and 

•  Direct  pressure  on  any  members  that  express  strong  arguments  against  any  of  group  stereotypes 
and  self-censorship  of  doubts  or  counterarguments  that  a  member  of  the  group  might  have  in 
order  to  create  an  illusion  of  unanimity  within  the  group  [4]. 

•  Other  types  of  bias  that  affect  groups  include: 

•  Imposing  imaginary  constraints  on  the  range  of  options; 

•  Sensitivity  to  reference  points  (e.g.  the  way  we  mentally  bracket  groups  of  decisions  together 
often  influences  how  much  risk  we  are  prepared  to  take  when  making  each  decision;  as  an 
example:  taking  a  broad  view  on  study  portfolios  can  act  as  an  antidote  to  excessive  risk 
aversion  when  dealing  with  individual  studies);  and 

•  Non-rational  escalation  of  commitment  (an  unwillingness  of  individuals  to  reverse  decisions 
if  they  feel  high  personal  responsibility  for  poor  outcomes  occurring  early)  [6]. 

Methods  for  reducing  the  effects  of  cognitive  bias  include  conducting  statistical  surveys  (essentially  in 
order  to  average  out  the  bias),  triangulating  several  sources  and  including  technical  or  ‘hard’  investigations 
in  order  to  confirm  statements  about  measurable  quantities.  When  addressing  group-think,  the  critical 
aspect  is  effective  facilitation  (ref.  Chapter  4  and  Section  7.3). 

7.2.2  The  Critical  Viewpoint 

Critical  analysis  of  sources  (be  they  humans  directly  in  person  or  indirectly  through  written  sources)  view 
humans  as  actors  with  a  possible  stake  in  the  situation.  This  view  essentially  includes  the  previously 
described  cognitive  viewpoint,  but  also  adds  that,  as  a  stakeholder,  the  person  may  have  an  interest  in 
concealing  facts  or  taking  a  certain  perspective  on  events  to  try  to  affect  outcomes  in  his  favour.  Important 
criteria  to  look  for  in  the  critical  analysis  of  any  source  are1: 

•  Authenticity  (i.e.  is  the  source  what  it  claims  to  be?); 

•  Concurrence  (i.e.  is  the  source  reasonably  concurrent  with  the  reported  events,  how  close  was  the 
person  to  the  events  they  are  describing?); 

•  Independence  (i.e.  are  two  seemingly  confirming  sources  actually  mutually  independent?);  and 

•  Bias  (i.e.  does  the  writer  have  any  reason  to  skew  the  message?)  [7]. 


1  Note  the  relationship  with  issues  discussed  in  Chapter  3. 
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Bias  in  this  perspective  is  no  longer  merely  the  unconscious  bias  that  comes  from  the  cognitive  shortcomings 
of  humans  as  outlined  in  Section  7.2.1,  but  also  the  bias  that  comes  from  an  actor  with  a  stake,  and  his 
possible  reasons  to  alter  or  conceal  available  information.  Within  the  critical  tradition,  it  is  important  to  try 
and  identify  what  the  bias  might  be,  and  how  the  bias  may  affect  the  data. 

Normally,  several  sources  are  used  to  conduct  an  investigation.  In  order  to  confirm  any  statements,  it  is 
important  to  know  whether  the  sources  are  actually  independent.  A  common  occurrence  is  so-called  ‘narrative 
contagion’  between  sources,  which,  for  instance,  occurs  when  witnesses  have  had  the  opportunity  to  confer. 
This  process  creates  a  bias  that  means  that  the  sources  can  no  longer  be  used  for  mutual  confirmation  of 
statements. 

Methods  for  conducting  critical  analysis  of  sources  are  primarily  to  assess  each  source  according  to  the 
criteria  for  critical  analysis  (authenticity,  concurrence,  independence  and  bias)  and  again  triangulating 
several  independent  sources  in  order  to  try  to  minimise  the  effects  of  the  limits  of  each  source. 

7.2.3  The  Constructive  Viewpoint 

The  two  previous  viewpoints  are  essentially  realist;  they  take  as  their  starting  point  the  axiom  that  there  is 
a  reality  ‘out  there’  that  can  be  discovered,  at  least  imperfectly  (as  with  critical  realism;  ref.  Section  3.3). 
On  the  other  hand,  the  constructive  viewpoint  takes  as  its  point  of  departure  that  reality  is  essentially 
socially  constructed.  ‘Truth’  in  this  context  is  the  best  informed  and  most  sophisticated  construction  on 
which  there  is  consensus,  though  this  definition  allows  several  constructions  that  meets  this  criterion  [8]. 
This  is  the  tradition  underpinning  qualitative  investigations  in  the  social  sciences. 

Our  starting  point  for  ‘soft’  OA  is  that  problems  are  socially  constructed;  thus  this  viewpoint  is  especially 
salient  for  problem  formulation.  In  the  constructive  view,  there  is  no  such  thing  as  an  independent  ‘fact’, 
facts  only  acquire  meaning  according  to  the  context  in  which  they  become  relevant.  Thus  ‘bias’  has  no 
absolute  meaning  in  the  constructive  viewpoint.  Each  stakeholder’s  view  is  essentially  equally  legitimate. 
This  could  be  construed  as  an  ‘anything  goes’  viewpoint,  but  the  process  of  consensus  acts  as  guarantor  of 
‘truth’  (or  some  truth  at  least),  and  thus  protects  to  some  extent  against  ‘anything  goes’.  In  the  context  of 
‘soft’  OA,  the  conclusion  is  that  those  stakeholders  that  meet  to  negotiate  a  problem  formulation 
essentially  create  their  own  new  ‘truth’  in  that  process,  to  the  extent  that  they  are  able  to  reach  consensus. 

The  main  challenge  when  conducting  an  investigation  from  a  constructive  viewpoint  is  ensuring  that  all 
key  stakeholders  are  given  a  ‘fair’  chance  to  air  their  view  and  take  part  in  the  construction  of  the  problem, 
and  that  there  is  an  equitable,  facilitated  process  that  allows  reaching  consensus. 

7.2.4  Data  Considerations  in  a  Multi-Methodology  Study 

Some  proponents  insist  that  the  constructive  and  realistic  paradigms  cannot  be  used  together,  but  experience 
shows  that  multi-methodology  investigations  work  well  in  practice.  This  is  also  supported  by  extensive 
literature  from  the  social  sciences  field,  proposing  the  pragmatic  position  that  considers  the  combination  of 
quantitative  and  qualitative  methods  perfectly  reasonable,  and  even  innovative  (for  instance  [9],  [10]  and 
[11])- 

While  the  viewpoints  as  discussed  in  Sections  7.2.1  -  7.2.3  may  differ,  the  approaches  for  ensuring  data 
quality  have  strong  similarities.  Triangulating  several  sources  is  therefore  desirable,  as  is  conducting  a  fair 
facilitation  process.  Triangulation  is  further  discussed  in  Section  8.2.1. 

Data  may  be  gained  from  the  field,  through  interpretation  of  procedures  (e.g.  manuals),  generated  by  models 
or  derived  by  artificial  environments  such  as  war-games  or  the  use  of  workshops  involving  SMEs  or  by 
approaching  experts  individually.  Data  has  been  described  as  ‘hard’  or  ‘soft’  depending  on  whether  it  is 
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respectively  derived  from  observer  independent  or  opinion/observer  dependent  sources  [12],  A  recent 
publication  [13]  has  proposed  that  field  data  may  be  collected  through  the  subjective  opinions  of 
stakeholders,  through  objective  measurements  or  even  from  informed  SMEs.  This  mix  of  subjective  and 
objective  data  is  a  key  aspect  of  ‘soft’  OA  and  should  be  regarded  as  a  strength  (because  of  its  richness) 
rather  than  a  weakness,  even  though  there  may  be  a  perceived  bias  [14]  towards  quantitative  data  measured 
in  a  ‘statistical  manner’.  Nevertheless,  the  sources  of  data  gained  by  different  means  should  be  well 
documented,  particularly  when  one  relies  on  subjective  opinion  and  informed  SME  speculation. 


7.3  WORKSHOPS,  MODELLING  AND  LACILITATION 

The  conduct  of  workshops  and  facilitated 
model  building  is  the  most  common  approach 
taken  by  many  ‘soft’  OA  methods  [15]. 

Understanding  the  main  strengths  and 
weaknesses  of  workshop  methods  is  thus  an 
important  issue  when  conducting  a  ‘soft’  OA 
study.  Workshops  are  mainly  used  to  provide  a 
venue  for  stakeholders  to  meet  in  order  to 
inform  each  other,  to  agree  on  problem 
formulations  and  to  create  courses  of  action. 


Schuman  and  Rohrbaugh  [16]  have  introduced  a  model  for  working  with  groups  (Table  7-12).  In  an  ideal 
group,  all  the  twelve  conditions  are  fulfilled.  The  model  can  also  be  used  for  diagnosis  if  difficulties 
appear  in  conducting  workshops.  In  a  situation  of  limited  resources,  the  facilitator  may  have  to  focus  on 
fulfilling  a  selected  sub-set  of  the  Schuman  conditions.  The  model  should  be  used  as  a  checklist  by  the 
analyst  and  facilitator  to  identify  the  key  conditions  that  will  ensure  that  the  major  obstacles  to  a  successful 
outcome  of  the  workshops  are  removed. 


In  a  series  of  workshops  addressing  new  operational 
concepts  for  maritime  mine  counter  measures,  participants 
sometimes  had  difficulty  with  keeping  to  agreed  definitions 
and  assumptions  (new  ideas  kept  popping  up).  Also  the 
change  of,  in  the  first  workshop,  evaluating  against  each  of 
several  specific  scenarios  to,  in  the  follow-on  workshop, 
evaluating  against  a  general  mine-hunting  setting  appeared 
confusing  for  some  and  posed  challenges  to  the  facilitator. 

If  a  study  requires  several  workshops  with  some 
participants  always  attending  and  others  not,  the  end-to- 
end  integrity  of  the  analysis  must  be  ensured. 


Table  7-1:  Conditions  for  Working  with  Groups  (Adapted  from  [16]). 


Factors 

Perspectives  4^ 

Group  Context 

Group  Structure 

Group  Process 

Relational 

Incentives  provided  to 
motivate  collective  work. 

Participants  are  sincere, 
open  to  each  others’ 
views  and  promote  trust. 

Conflict  is  managed. 

Political 

Resources  adequate  to 
complete  task. 

Legitimate  leadership 
present;  participants  have 
authority  to  make 
decisions  (up  to  a  point). 

Stakeholder  interests  well 
represented. 

Empirical 

Meeting  environment 
designed  to  foster 
productivity. 

Group  composition 
provides  needed  skills 
and  abilities. 

Modes  of  communication 
enhance  interactions. 

Rational 

All  relevant  data/artefacts/ 
expertise  is  available. 

Goals,  objectives,  tasks 
clearly  defined. 

Problem  exploring  and 
(possibly)  solving 
methods  used. 

A  central  requirement  for  conducting  successful  workshops  is  an  independent,  impartial  facilitator. 
The  facilitator  should  observe  roles  and  role  behaviour  and  needs  to  be  aware  of  the  power  and  politics 


2 

Their  circular  model  is  shown  as  a  table  in  the  CoBP. 
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that  may  affect  the  group3.  One  option  for  addressing  the  possible  bias  that  may  occur  due  to  power 
relations  between  the  group  members  is  to  deploy  a  method  that  allows  anonymous  data  entry,  such  as 
computer  supported  Journey  Making  [18].  Two  important  pitfalls  are  the  emergence  of  ‘group-think’, 
and  the  tendency  for  premature  concurrence.  The  facilitator  needs  to  initially  encourage  divergent  thinking 
in  order  to  avoid  group-think,  and  then  help  turn  the  group  towards  convergent  thinking  in  order  to  reach 
consensus.  The  job  of  the  facilitator  is  to  encourage  and  support  everyone  within  the  expert  group  to  do 
their  best  thinking.  In  addition,  the  facilitator  needs  to  have  enough  knowledge  about  the  subject  matter 
being  discussed  and  the  acronyms  being  used  during  the  workshop  so  that  he  can  keep  track  of  the  flow  of 
the  discussions,  ask  the  right  questions  and  effectively  guide  group  processes  and  procedures.  It  definitely 
helps,  but  it  is  not  absolutely  necessary,  for  the  facilitator  to  have  significant  content  related  knowledge. 
For  a  further  discussion  of  the  facilitator’s  role,  see  Chapter  4. 

Even  in  a  single  language  setting,  common  words  and  phrases  may  have  different  meanings  in  different 
organisations  or  services.  Hence,  in  order  to  avoid  mis-communication,  it  may  be  useful  to  create  a  lexicon 
for  participants  that  provides  standard  definitions  of  key  terms  used  within  the  ‘soft’  OA  study.  Further, 
facilitators  may  need  to  provide  assistance  to  participants  in  the  NATO  setting  who  do  not  have  a  good 
command  of  the  chosen  language  in  which  the  study  is  being  conducted,  e.g.  remaining  neutral  but  also 
helping  such  individuals  in  expressing  and  defending  their  viewpoints. 

In  most  ‘soft’  OA  methods,  a  model  of  some  kind  is  a  central  artefact  that  the  group  constructs  together. 
A  model  in  this  sense  is  a  representation  of  reality  built  for  some  definite  purpose  [19]: 

“The  model  should  ser\’e  as  an  aid  to  thinking  and  group  learning  but  will  not  produce  the 
right  answer.  The  model  should  be  ‘requisite’,  i.e.  sufficient  in  form  and  content  to  resolve 
the  issues  at  hand.  It  represents  the  collective  view  of  the  group  at  any  point  during  its 
generation  and  modification,  and  serves  as  a  means  to  examine  the  impact  of  differences  in 
perspective  or  vagueness  in  the  data.  Because  the  model  is  projected  for  all  participants  to 
see  it  as  it  is  created,  it  is  less  likely  to  be  perceived  by  participants  as  a  ‘black  box  ’,  which 
helps  to  gain  confidence  in  model  results.  ”  ([20],  see  also  Section  5.4). 

These  models  are  usually  qualitative  in  nature.  They  describe  the  various  concepts  that  are  used  in  the 
problem  formulation,  and  how  the  concepts  can  be  combined.  The  different  ‘soft’  methods  have  different 
approaches  and  styles  of  modelling. 

If  (parts  of)  the  problem  (are)  is  amenable  to  quantification,  a  qualitative  approach  can  be  supplemented 
with  quantitative  models.  Suitable  quantitative  modelling  environments  that  can  be  built  in  a  workshop 
setting  include  System  Dynamics  and  Bayesian  networks.  The  analyst  needs  to  be  aware  of  which  types  of 
problems  are  suitable  for  quantitative  treatment  in  a  workshop  setting  (see  also  [21]). 

The  facilitated  model  can  also  serve  as  a  test  bed  for  suggested  courses  of  action.  If  the  model  is  used  in 
this  fashion,  it  serves  as  an  aid  for  discovering  the  possible  consequences  of  action.  The  group  is  starting 
to  formulate  a  ‘theory  of  action’  for  possible  future  interventions  [22].  The  facilitator  can  assist  by  making 
this  theory  more  explicit.  On  occasion  the  theory  can  be  compared  with  established  knowledge  on  ‘what 
works’,  but  for  many  messy  problems  no  such  established  theory  exists.  The  group’s  theory  of  action  is 
essentially  an  assumption  or  a  hypothesis  on  what  may  work. 

As  we  have  seen  in  Section  6.3,  gaming  may  be  a  suitable  approach  when  the  problematic  situation  we  are 
facing  is  dominated  by  intentional  and  qualitative  uncertainty,  as  is  often  the  case  in  a  military  context. 
Like  workshops  and  facilitated  modelling,  gaming  is  (in  this  context)  intended  for  collective  learning 


In  the  context  of  military  decision  making,  the  main  client  is  normally  either  a  senior  officer  or  a  senior  Defence  official. 
In  addition,  military  procedures  tend  to  be  fonnal  and  traditional.  This  will  affect  how  a  problem  can  be  approached  and  what 
methods  may  be  socially  (i.e.  in  a  Defence  environment)  acceptable  [17]. 
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about  the  challenges  facing  stakeholders.  Gaming  should  not  be  seen  as  a  replacement  for  problem 
formulation,  but  as  a  continuation  of  the  exploration  of  options  for  addressing  the  challenges  ahead. 
Gaming  approaches  certainly  include  war-gaming  to  explore  tactical  and  operational  issues,  but  may  also 
include  other  forms  of  games.  Gaming  may  include  an  element  of  role-playing,  when  members  of  the 
gaming  group  assume  different  roles  in  order  to  explore  the  options  available  for  external  and  internal 
stakeholders,  and  normally  require  one  (or  several)  scenarios,  a  fictitious  setting  derived  from  the  general 
problem  statement.  Gaming  can  give  a  deeper  understanding  of  the  nature  of  the  problem,  and  the  options 
available  to  external  stakeholders  [23]. 

In  addition  to  some  well-established  methods  for  problem  structuring  in  a  workshop  setting  which  are  all 
usually  carried  out  in  a  facilitated  modelling  mode  ([1],  [15]),  another  methodology  that  can  be  useful  within 
a  workshop  setting  is  Multi-Criteria  Decision  Analysis  (MCDA).  This  family  of  methods  is  focused  on 
choice  in  a  situation  where  a  single  criterion  for  choice  between  options  does  not  exist,  and  on  a  structured 
approach  to  prioritising  between  multiple  criteria  [24].  The  weighting  and  scoring  activities  that  are  required 
pose  challenges  to  both  the  analyst  and  the  facilitator  when  dealing  with  the  different  preferences  that  usually 
exist  in  groups. 

Once  the  ‘right’  terms  are  identified,  objective 
errors  (measuring  the  thing  ‘right’)  appear  in 
addition  to  residual  subjectivity  issues.  Any 
term  ending  in  ‘  ility’  (such  as  mobility)  or  a 
similar  suffix  (such  as  ‘ness’  or  ‘ment’)  is 
likely  to  be  prone  to  this  problem.  Such  issues 
are  an  indication  of  the  appropriateness  of  a 
‘soft’  OA  approach.  Thus,  for  example, 
military  effectiveness  may  be  reasonably 
proposed  to  comprise  a  mix  of  subjectively 
weighted  terms  like  lethality,  mobility, 
survivability  and  sustainment.  Each  of  these 
derived  terms  is  challenging  to  define  and  is 
likely  to  be  composed  of  another  subjective  weighting  of  selected  surrogate  terms.  There  are  established 
means  to  treat  such  weightings  ranging  from  consensus  to  sophisticated  MCDA  approaches  [25].  Again  an 
audit  trail  and  agreement  are  required4. 

Proper  planning  and  preparation  for  the  group  session  is  critical  in  order  to  help  develop  participant 
confidence,  cooperation  and  ownership  of  the  session’s  products.  The  following  should  be  addressed  as 
part  of  the  planning  process:  session  objectives,  actual  products  of  the  session,  session  stages5  and  agenda, 
participants,  presentations,  reading  package,  consultation  and  expectation  management  (‘probable  issues’), 
record  keeping  arrangements,  and  organisational  practicalities.  The  reader  is  referred  to  [28]  for  further 
information  on  facilitating  workshops. 

The  quality  challenges  that  appear  in  problem  formulation  in  a  workshop  setting  are  more  or  less 
connected  to  the  fairness  of  the  process,  for  example: 

•  Were  all  key  stakeholders  considered? 

•  Was  the  process  accepted  by  the  involved  stakeholders? 

•  Does  the  problem  formulation  accurately  reflect  the  concerns  of  the  stakeholders? 


4  Although  scores  and  weights  of  ‘dissenters’  may  be  investigated  in  a  sensitivity  analysis,  rather  than  forcing  them  to  agree  to 
consensus. 

5  The  use  of  short  scripts  (‘thinklets’  [26],  or  a  ScriptsMap  framework  [27])  that  describe  and  visualise  distinct  activities  during 
a  session  may  be  helpful  to  get  the  purpose  of  these  activities  across  to  the  participants  and  create  a  desired  collaboration 
mode. 


A  study  involved  a  3-day  workshop  covering  the  design 
and  preliminary  evaluation  activities  regarding  alternative 
options  for  new  operational  concepts  for  maritime  mine 
counter  measures.  It  used  expert  opinion  in  addition  to 
previously  assembled  assessment  information  in  a  multi¬ 
methodology  analysis.  After  this,  a  more  technology- 
oriented  description  of  the  concepts  was  evaluated  in  a 
follow-on  workshop:  experts  assessed  the  technology  and 
its  impacts  on  the  operational  concepts  using  multi-criteria 
analysis. 

Presence  of  technical  (in  addition  to  operational)  expertise 
as  well  as  all  relevant  organisational  parts  of  the  Navy 
being  represented  proved  to  be  crucial  in  both  workshops. 


RTO-TR-SAS-087 


7-7 


DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 


ORGANIZATION 


These  challenges  are  met  by  adhering  to  process  and  facilitation  standards  as  indicated  earlier. 

Other  quality  challenges  are  added  when  options  for  action  are  considered,  for  example: 

•  Will  the  options  work  as  intended? 

•  Will  the  options  meet  their  objectives? 

•  Will  the  options  be  executable  within  reasonable  constraints  in  terms  of  time  and  resource  allocation? 

•  Will  the  options  be  acceptable  by  key  stakeholders? 

These  challenges  can  only  partially  be  met  within  the  framework  of  the  workshop  setting.  There,  these 
issues  should  be  addressed  by  ensuring  that  the  right  experts  are  consulted  and  that  the  assumptions 
underpinning  the  options  are  made  explicit,  including  the  ‘theory  of  action’6  behind  each  option.  If  the 
situation  is  characterised  by  dynamic  uncertainty,  designing  options  that  retain  freedom  of  action  and 
support  knowledge  generation  should  be  considered. 


7.4  OTHER  DATA  SOURCES  AND  SAMPLING  STRATEGIES 

It  may  not  be  possible  or  desirable  to  conduct 
all  required  data  collection  for  a  certain  study 
in  a  workshop  setting.  Other  data  collection 
approaches  may  be  necessary7.  Studies  of 
documents,  conducting  interviews  with 
stakeholders  and  eliciting  opinions  from  experts 
are  other  options  for  data  collection.  This  could 
include  experts  or  stakeholders  who,  for  some 
reason,  could  not  be  included  in  the  problem 
formulation  process,  but  could  be  consulted  in 
order  to  get  a  view  from  outside  the  core  group 
conducting  the  problem  framing. 

When  working  with  judgement-based  data 
collection,  it  is  important  to  have  a  purposeful 
sampling  approach,  i.e.  to  know  why  certain 
sources,  stakeholders  or  experts  are  chosen  for 
inclusion  in  the  study.  The  sampling  approach 
also  need  to  take  into  account  what  approach 
was  used  for  including  stakeholders  in  the 
problem  formulation  process  in  the  first  place. 

Options  for  qualitative  sampling  are  described  in 

Stakeholders  can  be  interviewed  in  order  to  get  an  understanding  of  their  view  on  an  issue,  or  to  gather 
deeper  knowledge.  Options  for  interview  styles  include  the  interview  guide  approach,  where  the  topics  of 
the  interview  is  identified,  but  the  actual  formulation  of  the  questions  takes  place  during  the  interview, 
and  the  standardised  open-ended  interview,  where  the  exact  questions  are  formulated  before  the  interview. 
Interviews  are  normally  conducted  one  to  one,  but  focus  group  interviews  are  also  an  option.  The  quality 
of  the  results  of  an  interview  is  largely  dependent  on  the  skills  of  the  interviewer  [9]. 


In  the  development  of  NATO  maritime  capability,  user 
requirements  for  monitoring  ‘white’  shipping  were  not  well 
understood.  To  improve  understanding,  a  cognitive  task 
analysis  was  conducted  using  critical  incident  inquiry  to 
identify  the  specific  cues  that  alert  operators  to  suspicious 
behaviour. 

Concept  mapping  was  employed  to  record  and  analyse 
operator  strategies  and  reasoning  as  they  worked  with 
systems  to  complete  tasks.  Performance  and  standard  work 
load  measures  were  also  collected  and  combined  with  the 
concept  maps  to  provide  an  overall  assessment. 

Expert  and  well  calibrated  observational  data  was  required 
for  reasonable  interpretation  of  the  way  operators  use 
strategies.  While  some  data  collectors/observers  were 
comfortable  with  the  tools  and  methods  being  used,  several 
were  only  available  for  two  days  before  the  execution  to 
train  and  exercise.  This  caused  some  difficulty  in  the 
analysis,  and  necessitated  a  much  greater  amount  of  post 
experiment  analysis.  Even  with  a  standard  method  for 
recording  observational  data,  collectors  need  to  have 
appropriate  time  and  training  to  be  familiar  with  the  relevant 
methods  and  theoretical  implications  associated  with  the 
behaviours  being  observed. 


Patton  [91. 


6  The  ‘theory  of  action’  [22]  (or  ‘theory  of  change’)  answers  the  question  “Why  do  we  think  this  course  of  action  will  reach  our 
objectives?”. 

7  Several  kinds  of  more  in-depth  quantitative  investigations  may  be  part  of  a  study,  but  data  quality  issues  for  quantitative 
investigations  are  not  considered  in  this  CoBP. 
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The  NATO  SAS-074  Activity  has  recently  devised  a  Human  Environment  Analysis  Reasoning  Tool 
(HEART).  The  tool  is  designed  to  enhance  access  to  data  and  methods  relevant  to  human  performance. 
‘Soft’  methods  are  often  the  only  techniques  through  which  human  aspects  can  be  addressed.  It  is 
recommended  that  ‘soft’  OA  analysts  should  access  the  tool  where  appropriate8. 


7.5  USING  SUBJECT-MATTER  EXPERTS 

The  opinions  of  SMEs  can  be  vital  to  a  ‘soft’  OA  study.  SMEs  may  be  included  in  workshop  sessions  or 
interviewed.  Another  option,  that  does  not  require  a  workshop  setting,  may  be  to  conduct  a  Delphi  study. 
The  SME  is  one  source  of  input  to  a  ‘soft’  OA  study.  While  the  SME  does  have  detailed  knowledge  it 
should  be  recognised  that  it  is  not  likely  to  be  complete,  could  be  biased  (whether  intentional  or  not)  and  is 
only  one  source.  Of  course,  unless  the  analyst  has  intimate  knowledge  of  the  system,  one  of  more  subject- 
matter  experts  should  be  involved  in  a  ‘soft’  OA  study  at  some  point.  Franco  and  Meadows  [29]  propose 
four  stages  where  SMEs  may  usefully  contribute,  with  this  CoBP’s  corresponding  phases  added  between 
brackets: 

*  Gathering  information  about  the  problematic  situation  (Appreciation  phase); 

*  Structuring  issues  (Analysis  phase); 

•  Designing  options  (Assessment  phase);  and 

•  Evaluating  options  (Assessment  phase). 

In  addition,  this  CoBP  proposes  an  Action  phase  (Chapter  5),  where  it  is  most  likely  that  SMEs  will 
contribute  too. 


As  a  study  participant,  the  responsibility  of  a 
SME  is  to  provide  his  informed  opinion,  based 
on  his  knowledge,  experience,  and  (technical, 
military,  social,  etc.)  expertise.  SMEs  are 
recognised  as  experts  by  their  peers;  they  may 
be  military  or  civilian.  For  example,  the  SME 
may  be  asked  to  provide  assumptions  regarding 
future  doctrine,  performance  data,  force  mixes, 
organisational  structures,  or  force  employment 
scenarios.  In  the  development  of  a  war-game, 
the  SME  could  contribute  by  proposing  system 
descriptions  or  by  developing  realistic  scenarios, 
and  during  its  execution,  the  SME  could 
consequently  play  the  role  of  sub-unit 
commander. 


In  NATO  a  capability  manager  was  asked  to  consider  the 
future  capabilities  that  might  be  needed  for  expeditionary 
operations.  He  took  the  NATO  definition  of  Expeditionary 
Operations  as  his  starting  point,  however  he  found  it  vague 
and  all-encompassing  and  therefore  not  useful  to  direct  or 
inform  capability  requirements  analysis.  Morphological 
analysis  was  identified  as  a  tool  to  obtain  a  more  detailed 
description  of  expeditionary  operations  that  could  be  used. 

One  issue  that  the  study  had  to  overcome  was  the  number 
of  experts  that  needed  to  participate  due  to  the  broad 
subject  area.  A  simple  electronic  data  collection  tool  was 
developed  to  collect  views  on  the  components  of 
Expeditionary  Operations  using  Morphological  Analysis  as 
a  framework.  This  tool  was  able  to  be  sent  out  via  email  for 
relevant  subject-matter  experts  to  complete  individually.  It 
was  also  used  as  a  data  capture  tool  in  a  series  of 
workshops,  allowing  flexibility  in  data  collection.  The  tool 
had  to  be  carefully  designed  to  allow  remote  collection  of 
qualitative  data. 


In  order  to  meet  his  responsibility,  the  SME  may  need  to  prepare  for  his  sessions  by  reading  study 
background  material  that  has  been  supplied  to  him.  SMEs  should  be  selected  in  a  way  that  helps  ensure  that 
experts  from  all  major  points  of  view  are  represented,  hence  helping  to  minimise  bias.  SMEs  may  be  selected 
by  relying  on  the  primary  client’s  knowledge  to  identify  SMEs  and/or  by  asking  known  SMEs  to  name  other 
SMEs.  However,  the  latter  may  result  in  a  lack  of  diversity  in  viewpoints.  The  number  of  SMEs  chosen  to 
participate  depends  on  the  nature  of  the  topic  being  addressed,  the  desired  type  of  interaction  between  the 


Draft  NATO/RTO  Technical  Report  (IST-999)  to  be  published  in  2011.  See  also  [a]  in  Section  7.7. 


RTO-TR-SAS-087 


7-9 


DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 


ORGANIZATION 


SMEs  (e.g.  face-to-face  group  session,  Delphi)  and  SME  availability  and  willingness  to  participate.  If  too 
few  SMEs  are  chosen  then  the  broadness  of  the  discussions  may  be  too  limited,  whereas  having  too  many 
SMEs  may  mean  that  not  everyone  has  an  adequate  opportunity  to  contribute. 

In  inviting  SMEs  to  participate  in  a  study,  information  needs  to  be  provided  on  the  study  aim,  the  identity  of 
the  sponsor,  what  tasks  the  SMEs  will  be  expected  to  perform,  and  how  much  time  it  is  expected  that  the 
SMEs  will  need  to  devote  to  the  study  and  over  what  period  of  time.  Some  of  the  real  experts  may  be  too 
busy  to  participate  at  the  required  level  of  effort.  They  may  change  their  minds  if  they  are  motivated  by  the 
opportunity  to  make  a  significant  contribution  to  the  issues  being  addressed  by  the  study  and  by  the 
recognition  such  contributions  may  result  in.  Rewarding  SMEs  for  their  services  may  slant  the  results  [30]. 

Expert  judgement  is  affected  by  the  process  of  gathering  it.  It  can  be  conditioned  by  how  questions  are 
phrased.  Experts  need  to  understand  what  they  have  to  do  in  the  working  group  session  and  why  (e.g.  how 
their  outputs  are  to  be  used)  and  how  they  are  expected  to  do  it.  There  needs  to  be  a  clear  and  common 
interpretation  of  terminology  being  used,  the  scales  on  which  judgements  (e.g.  ordinal  measurements)  are 
expressed  and  how  uncertainty  is  to  be  dealt  with  (e.g.  provision  of  estimates  as  a  range  of  values  rather 
than  a  point  value).  Facilitators  may  need  to  deal  with  situations  in  which  there  is  wide  disagreement 
between  experts’  judgements  or  in  which  there  is  a  seeming  consensus  in  the  judgements  except  for  one 
significant  outlier.  Study  results  will  be  criticised  if  there  is  the  perception  that  there  was  a  skewed  choice 
of  experts  or  if  irregularities  are  perceived  in  the  elicitation  and  analysis  process.  Readers  interested  in 
more  detail  on  the  elicitation  of  (quantitative)  judgements  are  referred  to  one  of  the  books  on  knowledge 
elicitation  (e.g.  [30]). 

As  for  any  ‘soft’  OA  study,  circumstances  will  dictate  where  and  at  what  stage  of  the  study  an  SME  could 
be  used.  In  this  respect,  the  situation  for  ‘soft’  OA  will  most  likely  be  different  from  that  for  a  ‘harder’  OA 
where  there  is  more  certainty  of  the  issues,  data  and  nature  of  the  study  on  the  outset.  For  this  reason  early 
involvement  of  SME  is  often  recommended  in  optimisation-biased  OR  textbooks.  The  pros,  cons  and 
requirements  of  SMEs  are  summarised  in  Table  7-2  in  terms  of  the  ‘4  A-phases’  model  described  in 
Chapter  5. 
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Table  7-2:  Pros  and  Cons  of  Including  SMEs  in  Different  Phases  of  the  Study. 


Stage 

Pros  of  Involving  a  SME 

Cons  of  Involving  a  SME 

SME  Requirement 

Appreciation 

Detailed  inside  knowledge. 

Could  be  tasked  to  obtain 
specific  information. 

Incomplete  knowledge  of  the 
system. 

Need  to  balance  analyst  team 
contribution  against  SME. 

In  depth  knowledge  of  how  the 
system  should  operate  and  its 
actual  behaviour. 

Ability  to  work  with  analysts. 

Analysis 

Gives  peer  review. 

Provides  sources  of  data  to 
test  the  model. 

Provide  personal  experience 
on  current  deficiencies. 

May  provide  useful 
speculation  on  new  ideas. 

Might  bias  the  rest  of  the 
study  through  a  personal  or 
institutional  mind  set. 

Formulation  of  a  model  that  is 
too  literal  with  little  opportunity 
to  explore  innovative 
changes. 

Embarrassment  to  the  study 
team  if  the  model  is  poor  and 
this  is  the  first  time  the  SME 
sees  it. 

May  be  reluctant  to  provide 
useful  speculation  on  new 
ideas. 

Acceptance  of  another’s 
perception  of  his  system. 

Ability  to  work  with  analysts. 

Ability  to  generalise  on  the 
nature  of  data  and  its 
applicability. 

Ability  to  speculate  on  ‘out  of 
the  box’  interventions. 

Ability  to  work  with  analysts. 

Assessment 

May  identify  the  study  team’s 
unknown  obstacles  to 
change. 

May  apply  their  own  filters  if 
the  changes  do  not  fit  their 
value  systems. 

Ability  to  provide  an  honest 
sanity  check. 

Ability  to  work  with  analysts. 

Action 

Provides  credibility  to  the 
study.  Identifies  pragmatic 
courses  of  action. 

Would  probably  have  to  be 
involved  in  most  of  the 
process  -  time  constraint. 

Needs  to  speak  with  authority 
and  be  recognised  as  such  by 
the  executive  decision  makers. 
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Chapter  8  -  INTERPRETING,  USING  AND  COMMUNICATING 

RESULTS  AND  OUTCOMES 


'  Chapter  1  ■  INTRODUCTION 

'  Chapter  2  •  PROBLEMATIC  SITUATIONS  AND  ‘SOFT  OA 
■  Chapter  3 -ACHIEVING  VALIDITY,  CREDIBILITY  AND  ACCEPTANCE 


•  Chapters  ROLESAND  RESPONSIBILITIES 


•  Chapters  -  THE  'SOFT  OA  STUDY  PROCESS 

•  Chapter6-  DESIGNING  THE ‘SOFT  OA  STUDY  METHODOLOGY 

•  Chapter  7  -  DATA  COLLECTION,  EXPERTS  AND  FACILITATED  WORKSHOPS 

•  Chapter  8  -  INTERPRETING,  USING  AND  COMMUNICATING  RESULTS  AND  OUTCOMES 


•  Chapter  9  -  THE  ESSENCE  OF  THE  CODE  OF  BEST  PRACTICE 

•AnnexA  -  LIST  OF  KEY  STATEMENTS  AND  RECOMMENDATIONS  OF  IMPORTANCE 
TO  THE  ANALYST 

•  Annex  B  -  SELECTIVE  LIST  OF  ‘SOFT’  METHODS 


>  Document  planned  method(s)  to  provide  a  reference  for  later  scrutiny  of  results. 

>  Conduct  triangulation  of  judgement,  using  documentation  and  empirical  data  where  possible  to 
increase  confidence  in  and  evidence  for  judgement. 

>  Determine,  in  co-operation  with  the  client,  where  bias  should  be  controlled  and  where  it  should  be 
measured  to  improve  understanding. 

>  Use  feedback  as  a  tool  to  understand  sensitivity  and  vested  interests. 

>  Give  careful  consideration  to  the  perceived  independence  of  the  analyst. 

>  Develop  a  clear  lexicon  by  which  outputs  of  the  study  can  be  described  to  the  client  community. 

>  Where  possible  include  the  client  or  his  stake  in  the  study  process  in  order  to  generate  ownership  of 
the  outcomes. 

>  Consider  how  learning  from  the  study  will  take  place  iteratively,  and  provide  outputs  that  support  this. 

>  Provide  capture  and  reporting  as  necessary  to  provide  traceability  and  transparency  in  the  methods 
used. 
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8.1  INTRODUCTION 

This  chapter  will  address  issues  concerning  the  results  and  outcomes  of  a  study  employing  ‘soft’  methods: 
inteipretation,  usage  and  communication.  Their  credibility  and  acceptance  depend  to  a  large  extent  on  the 
study’s  elements  that  have  been  discussed  in  previous  chapters  (e.g.  the  study  plan)  but  also  depend  on 
how  results  are  dealt  with  and  outcomes  are  communicated.  Triangulation,  sensitivity  analysis  and  coping 
with  bias  all  contribute  to  the  validity  of  the  results  and  will  be  discussed  in  this  chapter  (partly  again  but 
from  a  fresh  perspective).  The  final  sections  will  address  how  to  communicate  and  report  on  the  study, 
not  only  its  results  and  outcomes  but  also  how  it  was  conducted  as  this  contributes  to  its  traceability  and 
transparency,  and  thereby  its  auditability.  Refer  to  Section  1.6  or  the  Glossary  for  definitions  of  the 
concepts  of  result  and  outcome  of  a  study. 


8.2  INTERPRETING  THE  RESULTS  OF  A  ‘SOFT’  OA  STUDY 

Interpretation  can  be  defined  as  ‘to  explain  or  identify  the  meaning  of 1  ’  (the  results  of  an  OA  study). 
The  interpretation  of  the  results  of  a  piece  of  analysis  will  primarily  be  conducted  based  on  the  original 
objectives  of  the  study.  However  when  considering  a  study  based  predominantly  on  human  judgement, 
a  number  of  considerations  raise  themselves  when  interpreting  results. 

A  method  used  for  a  ‘soft’  OA  study  may  try  to  generate  consensus,  find  common  understanding  or  identify 
different  distinct  viewpoints  on  a  topic  of  interest.  Despite  this  there  is  the  scope  for  different  cognitive 
interpretations  of  the  results  from  the  study,  based  on  the  different  personalities  of  stakeholders  and  their 
organisational  and  political  stakes  in  the  problem  being  investigated.  It  is  therefore  beholden  on  the  analyst 
to  utilise  other  evidence  that  may  test  competing  judgements  and  provide  where  possible  the  most  robust 
interpretation  or  interpretations.  This  can  be  done  through  the  use  of  a  technique  known  as  ‘triangulation’. 

8.2.1  Triangulation 

When  planning  an  operational  analysis  study  more  than  one  method  can  be  employed  in  dealing  with  a 
problem.  When  using  different  methods  some  approach  is  required  to  synthesise  the  results.  This  process 
is  known  as  triangulation,  and  can  also  be  used  to  combine  qualitative  and  quantitative  results. 

Patton  [1]  identifies  four  types  of  triangulation: 

•  Data  Triangulation  -  The  use  of  a  variety  of  data  sources  in  a  study; 

•  Investigator  Triangulation  -  The  use  of  several  different  researchers  or  evaluators; 

•  Theory  Triangulation  -  The  use  of  multiple  perspectives  to  interpret  a  single  set  of  data;  and 

•  Methodological  Triangulation  -  The  use  of  multiple  methods  to  study  a  single  problem  or  program. 

Data  triangulation  will  be  familiar  as  a  tool  to  operational  analysts  (and,  in  fact,  to  all  scientists  and 
engineers),  the  nuance  being  in  this  case  that  the  data  in  question  will  primarily  come  in  the  form  of  human 
judgement.  This  then  identifies  the  need  to  consider  the  breadth  of  judgement  being  used  to  support  a  ‘soft’ 
OA  study. 

In  the  use  of  ‘soft’  OA,  where  judgements  are  being  collected  as  data,  investigator  triangulation  is  an 
important  and  useful  tool.  The  assessment  of  the  context  within  which  particular  observations  are  made  may 
change  from  observer  to  observer,  based  on  their  background,  experience  and  knowledge  of  the  subject 
under  study.  The  use  of  a  number  of  observers  or  evaluators  to  assess  a  particular  issue  is  important  for 
avoiding  bias.  A  further  possibility  is  to  conduct  investigator  triangulation  as  part  of  the  writing  of  the  final 
report,  where  the  various  observers  will  inform  the  deductions  and  conclusions  presented  to  the  client. 


1  Merriam-Webster  Dictionary. 
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Using  data  and  investigator  triangulation  is  a  process  of  applying  logic  and  asking  questions  such  as 
“Could  both  these  statements  be  true,  or  are  they  mutually  exclusive?”  While  triangulation  does  focus  on 
asking  questions  about  the  validity  of  data,  contradictions  do  not  mean  that  a  data  set  is  invalid.  Instead, 
contradictions  provide  greater  understanding  and  should  usually  be  the  subject  of  further  investigation 
( theory  triangulation). 

Methodological  triangulation  can  be  considered  when  designing  a  data  collection  plan,  i.e.  use  of  multiple 
methods  to  collect  a  particular  piece  of  data,  or  when  designing  a  study  as  a  whole.  The  benefits  of  a 
multi-method  approach  are  discussed  in  Chapter  6.  Theory  triangulation  may  actually  be  obtained  through 
method  triangulation  if  more  than  one  ‘soft’  method,  based  on  different  underpinning  theories,  is  used.  In 
the  case  of  PSMs,  theory  triangulation  may  be  an  integral  part  of  the  process,  as  the  desired  outcome  may 
be  an  understanding  of  the  various  perspectives  of  a  problem. 

When  an  analyst  is  looking  for  further  practical  evidence  that  may  substantiate  or  discredit  competing 
judgements,  he  has  two  further  sources  against  which  he  can  triangulate;  the  real  world  and  documented 
records  of  it.  If  an  opinion  about  a  subject  can  be  underpinned  by  discernible  observations  in  the  real  world 
then  it  may  carry  more  weight.  This  could  be  gauged  through  an  experiment  on  the  issue  in  question.  If  the 
judgement  cannot  be  underpinned  by  observations  in  the  real  world  then  supportive  documentation  may 
provide  greater  confidence  in  the  judgement.  This  could  be  in  the  form  of  historical  analysis,  weight  of 
professional  or  academic  opinion  or  documentation  of  relevant  policy,  process  and  standards.  In  the  military 
environment,  doctrine  or  standards  often  provides  a  useful  initial  comparator. 


8.3  SENSITIVITY  AND  BIAS 

This  section  will  discuss  two  other  aspects  of  using  judgement  (i.e.  sensitivity  and  bias)  that  could  challenge 
the  validity  and  thereby  credibility  and  acceptance  of  the  results  and  outcome  of  a  ‘soft’  OA  study. 

One  of  the  underpinning  aspects  of  conducting  OA  within  the  military  environment  is  that  the  analysis 
should  be  as  objective  as  possible  in  support  of  the  decisions  being  made  based  upon  it.  This  is  not  the 
same  thing  as  the  data  collected  being  objective  rather  than  subjective  in  nature.  Instead  it  means  the 
objective  analysis  of  subjective  data;  that  the  results  of  the  study  are  based  on  rules  grounded  in  theory  or 
established  practice  and  characterised  by  recorded  argumentation  and  rationale  and  following  an  agreed 
and  sound  process  accepted  by  all  involved,  and  that  these  results  are  presented  as  objectively  as  possible 
alongside  an  understanding  of  the  confidence  in  the  results,  their  sensitivity  to  particular  factors  and  the 
key  assumptions  undeipinning  them. 

‘Soft’  OA  based  on  human  judgement  should  be  treated  no  differently;  assumptions  made  should  be  tested 
and  thus  should  be  subjected  to  debate  and  sensitivity  analysis.  The  sensitivity  of  judgement  given  should 
be  captured  as  part  of  the  method  used.  There  are  a  number  of  practical  ways  and  methods  for  collecting 
data  on  the  sensitivity  of  human  judgement.  Often  these  are  based  around  asking  those  providing  their 
judgement  to  also  assess  how  confident  they  are  in  that  judgement  or  trying  to  capture  in  some  way  the 
consensus  or  diversity  of  opinion  about  a  particular  factor. 

As  such  one  of  the  most  useful  and  important  practical  tools  when  capturing  and  interpreting  judgement  is 
getting  those  participating  to  provide  feedback  on  their  judgements.  Some  ‘soft’  OA  methods  will  build 
this  in  as  part  of  the  process,  either  as  the  individual  providing  feedback  on  their  own  judgements  or  on  the 
judgements  of  the  group  as  a  whole:  a  good  example  is  the  Delphi  technique.  Feedback  by  participants, 
to  provide  sensitivity  on  judgements  given  should  be  built  into  any  study  relying  on  them. 

If  a  ‘soft’  OA  method  includes  quantification  of  some  aspects  of  judgement,  then  it  may  also  be  possible 
to  conduct  sensitivity  analysis  by  changing  weightings  or  factors  within  the  data.  A  good  example  of 
where  this  may  be  appropriate  is  Multi-Criteria  Decision  Analysis. 
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Another  traditional  underpinning  value  of  military  OA  is  that  analysis  should  be  independent  of  vested 
interests  (i.e.  ‘impartial’).  However  in  some  cases  when  using  ‘soft’  OA,  the  family  of  Problem  Structuring 
Methods  (PSMs)  in  particular,  this  may  not  be  wanted  or  the  case.  The  purpose  of  the  analysis  study  in  itself 
may  be  to  capture  and  better  understand  the  vested  interests  at  play.  This  is  often  the  case  when  trying  to 
structure  a  messy  problem  and  where  human  perception  defines  both  the  problem  and  the  potential  solutions 
available. 

In  light  of  this,  when  designing  a  ‘soft’  OA  study,  the  analyst  must  carefully  reflect  on  and  deal  with  bias. 
A  decision  will  need  to  be  made,  based  on  the  understanding  of  the  desired  outputs  of  the  study  with  the 
client,  where  vested  interests  and  bias  should  be  mitigated  and  where  it  should  be  deliberately  captured 
and  understood. 

The  analyst  should  understand  that  he  also  (alongside  those  also  involved  in  collecting  data  throughout  the 
analysis  study  such  as  the  facilitator;  ref.  Chapters  4  and  7)  will  introduce  bias  to  the  analysis.  When 
collecting  judgement-based  data  through  the  use  of  observers,  for  example,  it  is  important  to  triangulate  to 
avoid  the  views  of  one  observer  providing  a  potentially  biased  view. 

Finally,  in  some  ‘soft’  OA  studies,  and  particularly  in  PSMs,  the  perceived  stake  of  the  analyst,  be  it  by 
organisational  identity  or  otherwise,  may  lead  to  the  objectivity  of  the  analyst  being  questioned  by  some 
within  the  wider  client/stakeholder  community.  In  an  ideal  world,  data  collection  activities  such  as 
workshops  would  be  facilitated  and  analysed  by  people  truly  independent  of  any  party  within  the  client 
system,  however  in  reality  this  can  be  difficult  to  achieve. 

In  these  cases  participation  of  the  analyst  within  the  method  can  and  should  be  carefully  designed.  Some 
alternates  to  having  impartial  analysts  could  be  to  form  an  analysis  team  with  ‘trusted  representation’  by 
all  clients/stakeholders,  or  to  allow  a  number  of  reports  to  be  written  by  different  stakeholders  based  on 
one  set  of  data  and  analyses.  In  every  case  the  relationship  of  the  analyst  with  the  clients  and  other 
stakeholders  will  be  a  large  factor  in  both  understanding  and  dealing  with  perceptions  of  the  analyst’s 
impartiality. 


8.4  USING  AND  COMMUNICATING  RESULTS  AND  OUTCOMES 

Methods  based  on  judgement  are  quintessentially  concerned  with  communication  amongst  the  stakeholders. 
It  is  wise  to  ensure  that  the  plan  for  application  of  ‘soft’  methods  is  well  understood  by  participants  and 
clients.  The  same  goes  for  the  outputs  of  the  study  and  their  utility.  Good  use  should  be  made  of  opportunities 
for  debate  and  sensitivity  analysis,  preferably  with  the  involvement  of  all  stakeholders.  Whatever  happens, 
the  experience  of  stakeholders  will  be  enhanced  if  they  receive  feedback  on  their  involvement  and  they 
will  be  more  likely  to  contribute  further  to  both  the  current  analysis  and  further  studies.  When  reporting 
the  results  from  analysis  studies  there  are  two  primary  aims: 

*  Communicating  the  results  to  the  client  system;  and 

•  Providing  a  transparent  record  of  the  analysis  for  audit  purposes. 

The  methods  used  for  communicating  both  the  results  and  providing  a  record  of  the  study  will  differ  from 
case  to  case,  however  the  considerations  for  each  stemming  from  judgement-based  or  ‘soft’  OA  are 
highlighted  below.  If  the  primary  audience  for  output  is  policy  makers,  the  relevance,  clarity,  utility  and 
applicability  of  the  findings  will  be  of  the  utmost  importance. 

8.4.1  Communicating  Results  and  Outcomes  to  the  Client(s) 

One  of  the  differences  between  judgement-based  OA  and  ‘hard’  OA  is  that  the  results  are  primarily 
designed  to  improve  understanding  and  support  decision  making  rather  than  identify  a  unique  or  optimal 
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solution.  This  means  that  the  clients  and  other  stakeholders  (or  their  proxies)  will  need  to  be  willing  to 
participate  in  any  judgement  elicitation  process  and  must  understand  that  they,  not  the  analyst,  will  need  to 
own  the  model  and  the  results.  Inclusion,  or  at  least  representation,  of  the  client  base  is  a  prerequisite  for 
success  of  a  ‘soft’  OA  study.  It  also  means  that  the  outcomes  of  the  study  will  be  influenced  by  the  client 
and  will  not  be  generated  purely  by  the  analyst.  Politics  and  personalities  may  prove  to  be  significant 
factors  in  the  refinement  and  communication  of  outcomes. 

This  perception  of  the  validity  of  the  analysis 
amongst  the  stakeholders  (including  the  client 
community  and  other  participants)  will  be 
important  for  its  successful  use.  Its  acceptability 
for  use  in  supporting  a  decision  will  determine 
whether  it  is  seen  as  a  successful  piece  of 
analysis.  If  there  is  ‘expectation  failure’  with 
client  doubt,  whether  reasonable  or  unreason¬ 
able,  in  the  results  analysis  that  is  of  high 
quality  from  a  scientific  perspective  may  go 
unexploited. 


A  history-based  analysis  of  triggers,  causes  and  events 
allowed  forecasting  of  possible  consequences  of 
Australia’s  deployment  to  East  Timor,  in  particular  the 
effect  on  the  population.  While  the  intent  of  the  study  was 
to  identify  possible  situations  thus  allowing  ‘what-if  studies 
and  to  formulate  campaign  metrics,  there  was  a  notion  that 
the  results  could  be  used  to  predict  actual  (rather  than 
forecast  possible)  events  in  a  statistical  manner. 

Resolution  of  this  misinterpretation  was  however  readily 
achieved  after  explanation  of  the  role  of  a  ‘soft’  OA  study. 
This  is  an  example  of  how  too  much  may  be  made  of  ‘soft’ 
OA  studies,  particularly  when  it  comes  to  the  prediction  of 
likely  events. 


There  are  a  number  of  outcomes  from  a  ‘soft’  OA  study  of  which  the  analyst  needs  to  remain  cognisant 
([2],  [3]).  Each  will  have  different  import  for  the  different  stakeholders: 

•  The  model  built  of  the  situation,  which  contains  the  structuring  of  the  problematic  situation  and/or 
the  solutions  suggested.  The  model  will  be  used  for  performing  analyses  and  drawing  conclusions 
from  its  responses  in  the  study  itself,  but  will  also  have  potential  to  assist  in  the  analysis  of  other 
similar  situations  and  to  inform  later  decisions  about  the  system  under  study. 

•  The  communication  between  participants  providing  judgement  in  the  study.  ‘Soft’  OA  methods  have 
the  ability  to  allow  communicative  exchanges  between  stakeholders  that  are  more  comprehensible, 
legitimate  and  accurate.  With  a  structured  method  the  ability  to  process  information  is  improved 
allowing  participants  to  focus  more  on  the  content  and  their  arguments. 

•  Providing  increased  and  shared  understanding  of  the  problematic  situation.  This  is  often  the  real 
underlying  benefit  of  a  ‘soft’  OA  study.  The  participants  and  stakeholders  should  achieve  together 
a  better  understanding  of  the  problem,  potential  courses  of  action  and  others’  beliefs  and  positions 
about  the  situation. 

•  Strong  ownership  of  the  problem  formulation  and  the  actions  being  taken  to  address  it.  Improved 
shared  understanding  will  hopefully  lead  to  an  improved  common  purpose  in  moving  ahead, 
and  promote  ownership  of  the  problem  and  actions  being  taken. 

Although  the  model  and  the  results  of  the  study  will  likely  be  presented  and  clearly  visible  to  the  client  the 
other  outcomes  identified  in  the  bullet  list  above,  unless  articulated  by  the  analyst,  will  likely  remain 
invisible  (ref.  Figure  8-1).  The  benefits  of  communicating  and  shared  understanding  for  the  stakeholders 
and  participants  should  be  identified  and  clearly  articulated  by  the  analyst  alongside  the  direct  results  and 
conclusions  from  the  analysis.  Likewise,  the  analyst  should  ensure  the  study  method  and  process  enhance 
the  ownership  of  the  problem  and  solution  by  particular  stakeholders  as  well  as  articulate  it  directly  to  the 
client  system  when  appropriate. 
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•  Documentation  of  results  or 
options 

•  Record  of  analysis 

•  Action  plans  forthe  way 
ahead 


Common  conceptual 
models 

Inclusion  of  SMEs 
Commitments  to  further 
exploration 


¥  \ 

Outcomes 
of  'Soft'  OA 

b _ d 

•  Increased  and  shared 
understanding  of  the 
situation 

•  Mutual  appreciation  of 
the  way  ahead 

•  Improved  commitment  on 
way  ahead 


Comprehensible, 
legitimate  and  accurate 
communication 
Appreciation  of  others' 
values,  perceptions  and 
positions 


Figure  8-1:  The  Visibility  of  Outcomes  of  a  ‘Soft’  OA  Study  (Adapted  from  [4]). 


In  the  defence  sector,  where  ‘hard’  methods  have  been  used  for  many  years,  it  may  be  difficult  for  the 
client(s)  of  OA  to  understand  the  differences  in  product,  output  and  expectation  with  judgement-based  OA. 
In  fact  ‘hard’  and  ‘soft’  OA  may  be  understood  better  by  particular  clients  depending  on  their  background: 
an  education  in  the  physical  sciences  as  opposed  to,  say,  policy  analysis  or  business  studies  may  bias 
perspectives  of  utility  or  validity.  To  compound  a  lack  of  understanding  of  the  outputs  of  judgement-based 
OA  the  client  may  also  lack  confidence  in  the  same  analyst  to  deliver  those  outputs  as  results  from  a  more 
typical  ‘hard’  OA  study. 

The  consequence  of  all  of  this  is  that  the  analyst  will  need  to  be  careful  that  the  type  of  outcome  sought  by 
the  client  and  that  expected  to  be  provided  by  the  analyst  are  not  different.  The  analyst  will  need  to  explain 
clearly  to  clients  and  other  stakeholders  what  will  constitute  the  output  of  a  judgement-based  OA  study. 
As  discussed  in  Chapters  4  and  5,  the  analyst  will  need  to  include  the  client  in  the  definition  of  the 
problem  space  and  then  subsequent  (if  necessary)  iterations  of  the  problem  statement  and  options  so  that 
this  difference  in  expectation  is  avoided. 

One  of  the  difficulties  with  elicitation  of  judgement  and  the  analysis  based  on  that  elicitation  is  that  if  a 
stakeholder  has  not  participated  in  the  process,  he  does  not  have  the  experience  or  ‘buy  in’  for  the  results. 
Often  results  based  on  judgement  will  not  be  transferable  to  those  who  have  not  been  part  of  the  process, 
who  will  perceive  that  their  views  have  not  been  taken  into  account  and  distrust  the  results.  This  underscores 
the  importance  of  identifying  key  stakeholders  and  as  much  as  possible  including  them  in  the  judgement 
process,  from  problem  formulation  through  discussions  of  solution  options2. 


8.4.2  Reporting  and  Delivery 

Based  on  the  different  outputs  identified  above,  a  ‘soft’  OA  study  will  provide  usable  results  throughout  the 
study  as  well  as  at  the  end.  As  outlined  in  Chapter  5,  ‘soft’  OA  generally  follows  a  dynamic  and  iterative 
process  that  looks  to  improve  understanding  as  the  problem  space  is  investigated  further.  This  means  that 


2 

Sections  4.3.3  and  6.4  discuss  the  possible  ways  in  which  to  involve  stakeholders. 
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delivery  of  results,  through  communication  and  reporting  will  need  to  be  a  continuous  process  throughout  the 
study. 

The  iterative  nature  of  the  process,  however,  has  the  ability  to  cause  tension  with  clients.  Most  OA  studies  in 
the  defence  sector  will  be  part  of  a  wider  program  of  work  that  will  be  time  and  resource  limited  and  likely 
managed  using  a  standard  project  management  framework3.  To  make  ‘soft’  OA  results  credible  they  will 
need  to  produce  outputs  to  meet  deliverable  timelines.  The  analyst  will  need  to  describe  progress  clearly  to 
his  clients,  helping  to  map  the  outputs  of  the  ‘soft’  OA  study  to  the  wider  progress  of  the  client’s  work. 

Traditionally,  an  analyst  is  often  responsible 
for  producing  the  final  report  (if  agreed  to  as 
being  a  deliverable)  and  briefing  interested 
parties  on  study  recommendations  and  end 
products.  Although  formal  closure  and 
reporting  of  the  study  is  needed,  given  the 
nature  of  ‘soft’  OA,  it  is  likely  that  politically 
feasible  agreements  are  made  before  analysis 
is  complete  and  before  the  final  report  is 
finished.  It  is  also  likely  that  the  analyst  will 
need  to  provide  continued  support  to  the  client 
in  refining  and  delivering  the  identified 
outcomes  from  the  study. 


In  the  development  of  NATO  maritime  capability,  user 
requirements  for  monitoring  ‘white’  shipping  were  not  well 
understood.  To  improve  understanding,  a  cognitive  task 
analysis  was  conducted  using  critical  incident  inquiry  and 
concept  mapping  to  identify  operator  strategies  and 
reasoning  as  they  worked  with  systems  to  complete  tasks. 

A  turnover  in  the  leadership  of  the  project,  for  both  the 
client  and  the  lead  nation,  occurred  between  the 
completion  of  the  experiment  and  the  publication  of  the 
analysis.  This  happens  frequently  in  a  military  environment. 
This  resulted  in  a  loss  of  momentum  with  the  results  of  the 
experiment.  The  cognitive  task  analysis  and  concept  maps 
should  have  probably  been  the  most  enduring  parts  of  this 
work  (rather  than  tool  specific  recommendations),  but  were 
never  published  and  once  staff  had  rotated  they  were  less 
accessible  to  their  replacements. 


A  senior  decision  maker  will  usually  want  to  ‘deconstruct’  the  analysis,  in  the  sense  that  they  will  want  to 
understand  what  particular  features  of  the  alternative  options  and  scenarios  presented  are  giving  rise  to  the 
results.  They  may  then  wish  to  ‘repackage’  these  insights  to  reach  different  conclusions  about  the  preferred 
way  ahead  or  to  generate  different  outcomes.  These  may  differ  from  those  that  the  analyst  or  a  lower-level 
customer  might  have  identified,  not  least,  of  course,  because  the  more  senior  the  decision  maker  the  more 
strategic  the  viewpoint  tends  to  be  and  the  more  options  become  available. 

It  is  quite  right  that  decisions  should  be  taken  at  an  appropriate  level;  it  is  not  for  the  analyst  to  usurp  a 
policymaker’s  own  role  in  using  judgement  in  their  decision;  but  neither  should  he  simply  report  the 
results  of  his  analysis  in  a  mechanistic  way  and  wash  his  hands  of  any  interpretation.  The  analyst’s  task 
when  delivering  a  study  should  therefore  be  to  provide  conclusions  in  a  way  which  allows  decision  makers 
to  carry  through  the  thought  processes  in  a  structured,  auditable  way  and  which  inter  alia  facilitates: 

•  Identification  of  the  issues  that  have  been  resolved  by  the  analysis  and  how  they  interact  with 
those  that  have  not; 

•  Isolation  of  the  key  judgements  that  have  been  made  and  the  factors  on  which  they  depend; 

•  Resolution  of  ‘what-if  questions  in  real  time;  and 

•  Negotiation  of  a  rational  path  through  a  complex  problem  to  a  final  decision. 

Theoretically  from  a  systems  perspective,  even  after  action  is  taken,  the  flux  of  events,  ideas,  learning  and 
the  need  to  cope  continues  indefinitely.  It  is  for  this  reason  that  the  intangibles  produced  during  the  study 
continue  to  provide  value  even  after  the  study  is  formally  ended.  When  considering  the  output  and  reporting 
of ‘soft’  OA  an  analyst  should  consider  how  they  can  support  this  further  organisational  learning,  through  the 
presentation  of  models  and  data  and  pursuing  strong  ownership  of  the  way  ahead. 


3 

For  example:  Projects  In  Controlled  Environments  or  PRINCE2  as  used  by  the  UK  government  http://www.prince-officialsite. 
com/. 
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The  typical  models  and  subjective  data  that  come  from  a  ‘soft’  OA  study  may  also  be  considered  more 
difficult  to  present  and  understand  than  some  of  the  outputs  of  ‘hard’  OA  studies.  Diagrammatic  outputs, 
such  as  rich  pictures,  cognitive  maps  and  causal  maps  may  not  have  been  seen  before  by  a  client, 
and  without  participation  in  building  them,  may  be  difficult  to  grasp  and  understand.  Furthermore  over 
time,  the  utility  and  understanding  in  these  products  may  decrease  as  those  who  worked  on  them  are 
increasingly  unavailable  to  describe  them  first  hand.  These  are  considerations  when  reporting  results  that 
are  expected  to  have  use  for  a  prolonged  period.  The  analyst  will  also  need  to  consider  ways  of  dealing 
with  qualitative  data,  such  as  grouping  by  theme  and  colour  coding  outcomes. 

8.4.3  Traceability  and  Transparency 

Judgement-based  analysis,  just  as  with  any  form  of  OA  should  be  trustworthy.  The  analyst  should  aim  to 
be  as  independent  and  objective  as  possible  whilst  results  should  be  reported  with  no  undeclared 
assumptions  or  unstated  simplifications  to  the  model  developed.  As  a  member  of  a  scientific  community, 
analysts  have  an  obligation  to  monitor  and  report  their  own  analytical  processes  so  that  they  may  be 
subjected  to  scrutiny. 

In  contrast  to  scrutiny  of  ‘hard’  OA,  the  reason  for  transparency  in  judgement-based  OA  may  not  only  be 
so  that  results  are  repeatable  -  although,  in  theory,  this  is  a  desirable  goal.  Indeed,  working  with  the 
inherent  variability  of  opinions  and  judgements  of  humans,  there  will  be  limited  ability  to  repeat  a  method 
with  a  different  group  of  experts  and  produce  the  same  result4.  However,  the  analysis  methodology  should 
still  be  documented  alongside  the  results  so  that  an  external  scrutineer  can  understand  the  methodological 
framework,  and  what  it  aimed  to  achieve.  This  will  help  to  demonstrate  the  legitimacy  of  the  processes 
undertaken  during  the  analysis  as  well  as  the  legitimacy  of  the  results.  Methods  used  should  be  shown  to 
be  based  on  solid  theoretical  and  philosophical  grounds.  Furthermore,  a  list  of  the  participants  and  SMEs 
who  have  provided  judgement  to  support  the  study  should  be  captured  and  reported.  This  will  help  those 
who  wish  to  scrutinise  the  study  to  understand  where  data  in  the  form  of  judgement  has  come  from. 
Although  it  may  be  inappropriate  to  report  the  names  of  individual  participants,  information  about  their 
expertise  or  organisational  stake  should  be  recorded  and  presented. 

A  sound  elicitation  of  judgement  should  therefore  be  preceded  by  thorough  preparation,  including  the 
building  of  a  sound  analysis  plan.  This  should  include  considerations  for  dealing  with  bias,  being  able  to 
conduct  sensitivity  analysis  and  building  confidence  in  the  credibility  and  acceptability  of  the  results. 
Practical  ways  of  doing  so  are  explored  further  below. 

The  methods  selected  and  the  level  of  validation  needed  will  depend  on  the  level  of  risk  and  importance  of 
the  decision  being  supported.  There  is  always  the  need  for  this  to  be  agreed  and  understood  between  the 
analyst  and  the  client.  When  conducting  judgement-based  analysis,  particularly  when  using  PSMs  it  should 
be  recognised  that  this  may  not  be  clear  at  the  beginning  of  the  analysis  process,  in  fact  it  is  something  that 
will  have  to  be  iteratively  developed  along  with  the  exploration  of  the  problematic  situation. 

The  documentation  of  an  analysis  plan  also  plays  another  key  role.  When  looking  at  the  validity  of  the 
results,  particularly  of  PSMs,  it  provides  a  reference  for  scrutiny  purposes.  The  plan  outlines  in  advance 
the  methodological  framework,  in  terms  of  which  something  will  be  defined  that  counts  as  ‘knowledge  to 
be  gained  from  the  analysis’.  Through  this  an  external  person  can  recover  the  mental  processes  used 
throughout  the  analysis  which  led  to  the  conclusions  reached,  helping  to  prevent  different  interpretations 
of  the  results  by  a  scrutiniser  and  the  analyst. 

A  good  plan  will  address  most  of  the  issues  identified  in  the  preceding  chapters  of  the  CoBP.  The  plan 
should  also  recognise  the  potential  for  introducing  biases  into  the  analysis  process,  and  identify  how  to 


4  This  may  rather  be  regarded  as  a  strength  of  ‘soft’  OA:  repetition  with  different  groups  leading  to  different  results  may  open 
up  and  even  extend,  for  example,  the  option  space. 
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understand  them  if  not  how  to  control  them  (they  may  be  a  fundamental  part  of  the  outcome  wanted  in  a 
‘soft’  OA  study). 

Practically,  the  traceability  of  the  analysis  will  normally  be  provided  by  a  full  and  detailed  study  report. 
Other  tools  that  can  help:  an  analyst  can  keep  a  logbook  of  their  progress  with  judgement-based  analysis 
just  as  they  might  when  running  a  simulation.  This  can  be  used  to  note  observed  bias,  political  viewpoints 
as  well  as  the  iterative  development  of  both  problem  and  solution  space.  As  with  other  OA  studies,  results 
and  method(s)  can  be  exposed  to  peer  group  professionals  through  conferences,  journals  and  papers. 
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Chapter  9  -  THE  ESSENCE  OF  THE  CODE  OF  BEST  PRACTICE 

This  chapter  concludes  the  analyst-oriented  volume  of  the  Guide  and  summarises  its  essential  points. 

Three  key  aspects  characterise  a  judgement-based  (‘soft’)  OA  study: 

•  It  enables  progress  to  be  made  for  some  otherwise  intractable  and  complex  decisions. 

•  It  involves  a  creative  journey  of  discovery  and  learning  that  can  be  used  to  the  advantage  of  decision 
makers. 

•  The  inherent  uncertainty  of  complicated  decision  situations  that  the  defence  sector  faces,  leads  the 
client  for  judgement-based  OA  to  what  are  perhaps  his  most  pressing  concerns  -  its  validity, 
credibility  and  acceptance.  Study  methods  must  therefore  be  well  documented  to  withstand  scrutiny. 

Judgement-based  OA  is  used  to  inform  decisions  where  the  subject  matter  is  incompletely  known  or 
understood,  where  many  and  possibly  conflicting  viewpoints  are  involved  and  there  is  initially,  at  least, 
no  obvious  single  solution.  Many  real-world  and  military  issues  can  be  described  in  this  way.  The  term 
‘soft’  is  used  in  the  academic  community  to  denote  such  OA  studies  as  often  much  of  the  input  and  output 
comprises  subjective  information.  These  issues  may  be  compared  to  cases  where  most  of  the  issues  of  a 
problematic  situation  are  known  and  there  is  a  logical  process  to  derive  more  quantitative  ‘hard’  insights. 
In  reality  there  is  no  rigid  distinction  and  most  analyses  comprise  both  ‘hard’  and  ‘soft’  elements. 

An  easy  way  to  distinguish  between  the  two  is  to  note  that  ‘hard’  OA  is  more  likely  to  give  checkable 
facts  whereas  ‘soft’  OA  is  predominantly  based  on  judgement  (i.e.  informed  opinions)  and  more  likely  to 
lead  to  a  body  of  evidence  allowing  exploration  and  comparison  of  possibilities.  Many  real-world  issues 
lie  in  the  latter  category:  particularly,  high-level  questions  relating  to  strategy,  capability  development  and 
major  acquisitions. 

It  should  also  be  noted  that  a  judgement-based  approach  may  still  include  some  ‘hard’  or  quantitative 
elements  (ref.  bottom  boxes  of  Figure  3-1);  this  is  known  as  ‘complementarity’.  It  is  often  the  case  that  the 
analyst  needs  to  dig  deep  into  a  problem,  for  instance,  to  test  the  feasibility  of  adopting  a  particular  option. 

Chapter  2  of  this  Code  contains  a  table  (Table  2-2)  that  characterises  ‘soft’  and  ‘hard’  OA  and  suggests 
which  is  more  suitable  for  different  types  of  problematic  situations.  Chapter  6  contains  a  list  of  questions 
(checkboxes;  Section  6.2)  that  helps  determine  the  nature  of  a  problematic  situation.  The  decision  on  whether 
to  take  a  typical  judgement-based  approach  should  properly  be  left  to  the  analyst  after  due  discussion  with  the 
client. 

This  volume  of  the  Guide  is  focused  for  the  benefit  of  an  analyst  on  the  concept  and  structure  of  a  ‘soft’ 
OA  study  and  its  methods  and  on  how  to  achieve  rigorous  and  auditable  quality  control  of  both  process 
and  content.  However,  good  practice  requires  that  both  the  analyst  and  client  are  aware  of  their  roles  and 
responsibilities  regarding  the  study  and  that  judgement-based  OA  is  in  many  ways  a  collaborative  journey 
involving  subject-matter  experts,  clients  and  analysts. 

The  analyst  is  the  expert  in  the  design,  development  and  implementation  of  the  study  plan.  He  should  be 
guided  by  established  practice,  previous  work  and  experience  and  his  general  skills.  Part  of  his  expertise 
lies  in  knowing  what  steps  to  take  to  ensure  that  a  sound  process  has  been  followed  in  two  respects: 

•  Was  the  approach  taken  appropriate  for  the  maturity  of  the  issue?  ( Was  the  right  study  done?) 

•  Was  due  diligence  employed  in  carrying  out  the  study?  (Was  the  study  done  right?) 

The  good  ‘soft’  OA  analyst  will  have  many  attributes  and  skills  to  bring  to  problematic  situations.  He  will 
also  be  responsible  for  assembling  an  analytical  team  comprising  the  appropriate  skill  sets  and  access  to 
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computers  and  other  resources.  Chapter  4  discusses  the  skills  needed  by  a  good  ‘soft’  OA  analyst, 
including  a  summary  of  his  responsibilities  (Section  4.5). 

As  has  been  discussed  in  the  CoBP,  the  validity  of  a  judgement-based  OA  study  is  hard  to  assess. 
Similarly,  it  is  difficult  to  define  in  advance  a  formal  pattern  of  analytical  work.  Thus  a  flexible  approach, 
allowing  for  review  and  iteration,  is  preferred  to  a  rigid  mechanism  where  every  step  has  to  be  completed 
before  the  next  can  be  started.  Chapter  5  discusses  this  iterative  and  divergent/convergent  nature  of  a  ‘soft’ 
OA  study. 

The  CoBP  has  discussed  key  activities  to  be  conducted  by  the  analyst  and  has  suggested  ways  of 
conducting  them  so  as  to  maximise  the  study’s  validity.  It  is,  of  course,  recognised  that  individual  Nations 
will  chose  to  implement  the  recommendations  of  the  Code  in  different  ways.  For  instance,  some  parts  may 
be  strengthened,  others  ignored.  It  is,  however,  to  be  noted  that  documentary  record  of  the  design  process 
would  provide  good  primary  evidence  of  validity  for  any  judgement-based  study  that  was  likely  to 
generate  contentious  debate,  such  as  for  major  acquisitions.  This  should  not  indicate  simply  that  every  step 
was  completed,  but  rather  that  each  one  was  reflected  upon  with  reasons  given  where  it  was  not  followed. 

It  needs  to  be  remembered  that  a  study  outcome  is  only  one  input  into  a  decision-making  process. 
For  more  quantitative  OA,  these  decisions  may  be  easy  to  make  as  there  is  clear  numerical  evidence 
presented  for  a  well-understood  issue.  For  judgement-based  OA  neither  the  initial  issue  is  well  understood, 
nor  are  the  results  of  the  study  usually  clear  cut.  These  circumstances  can  be  seen  in  both  a  positive  and 
negative  light,  leading  to  the  identification  of  the  benefits  of,  and  threats  to,  a  ‘soft’  OA  study.  The  benefits 
are  set  out  in  Table  9-1. 


Table  9-1 :  Benefits  of  a  ‘Soft’  OA  Study. 


•  The  ability  to  analyse,  make  progress  on,  and  perhaps  resolve,  problematic 
situations  that  would  otherwise  remain  intractable. 

•  An  improved  and  more  widely  shared  understanding  of  issues  and  solutions. 

•  An  improved  sense  of  common  purpose  and  greater  commitment  to  ways 
forward. 

•  The  discovery  of  alternative  options  for  dealing  with  an  issue. 

•  The  iterative  development  of  acceptable  ways  forward. 

•  The  systematic  gathering  and  analysis  of  relevant  information  and  knowledge. 

•  A  better  appreciation  of  different  objectives,  perspectives  and  values,  and  the 
ability  to  reconcile  them. 

•  The  delivery  of  usable  products  throughout  the  study. 


All  of  these  items  contribute  to  the  development  of  a  shared  common  picture  of  the  issues,  even  if  there  is 
disagreement  on  detail.  They  add  value  to  the  client  and  allow  an  informed  climate  of  debate.  If  these 
items  are  conducted  in  the  best  possible  way,  that  value  is  likely  to  be  maximised. 

The  threats  to  a  ‘soft’  OA  study,  whether  intentional  or  unconsciously  directed  by  individuals’  decision 
making  preferences  or  backgrounds,  are  set  out  in  Table  9-2. 
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Table  9-2:  Threats  to  a  ‘Soft’  OA  Study. 


•  Disagreement  with  a  specific  part  of  the  study  leading  to  a  dismissal  of  the  rest  of 
the  material. 

•  Treating  ‘soft’  OA  study  results  with  too  much  certainty  (e.g.  as  a  prediction). 

•  Too  rapid  a  progression  from  an  ill-formed  concern  to  a  rigid  plan  for  change 
(e.g.  an  acquisition). 

•  Biased  preference  of  some  forms  of  evidence  (e.g.  quantitative  sources). 

•  Selective  interpretation  to  support  a  specific  argument. 


It  is  the  key  task  of  both  the  analyst  and,  if  present,  the  facilitator  to  recognise  and  minimise  the  effect  of  the 
threats.  If  done  properly,  this  will  also  add  to  the  study’s  validity  and  thereby  credibility  and  acceptance. 

In  summary,  the  analyst  should  work  with  the  client  to  ensure  that  an  appropriate  degree  of  validation  has 
been  achieved  by  a  judgement-based  OA  study,  particularly  when  SMEs  or  other  stakeholders  may  bring 
information  into  the  study  at  any  time  (Chapters  6  and  7).  This  cooperation  is  of  crucial  importance  through 
the  entire  cycle  of  the  study  in  order  to  generate  ownership.  The  CoBP  addresses  how  the  approach  to  a 
study,  and  the  cooperation  with  the  stakeholders,  contribute  to  validity.  Chapter  3  in  particular  discusses  the 
general  philosophy  of  achieving  validity  (Figure  3-1).  Chapter  4  addresses  the  responsibilities  that 
stakeholders  (including  clients)  may  discharge  in  this  matter. 

Finally,  Annex  A  collects  together  the  key  statement  boxes  which  open  each  chapter.  Reflecting  upon  the 
key  statements  and  exploiting  the  checklists  is  one  possible  use  of  this  CoBP  in  seeking  to  achieve  valid 
and  thereby  credible  and  accepted  study  outcomes. 
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>  Key  statement  (to  be  kept  in  mind) 

□  Recommendation  (to  be  checked  and  ticked) 

A.l  CHAPTER  2  -  PROBLEMATIC  SITUATIONS  AND  ‘SOFT’  OA 

>  Three  types  of  a  problematic  situation  can  be  distinguished:  puzzle,  problem  (proper),  mess.  Each  type 
is  approached  somewhat  differently.  ‘Soft’  OA  tends  to  be  more  suitable  for  problems  (proper)  and 
messes. 

>  ‘Hard’  and  ‘soft’  OA  both  support  decisions  regarding  problematic  situations.  There  are  however 
numerous  differences  between  them  from  a  large  number  of  perspectives,  although  in  practice  each 
perspective  represents  a  spectrum. 

>  Uncertainty  is  one  of  the  key  phenomena  that  need  to  be  addressed  in  a  problematic  situation  as  it 
characterises  a  problematic  situation  to  a  large  extent. 

>  ‘Soft’  OA  aims  to  structure  problematic  situations  (with  a  focus  on  problems  proper  and  messes), 
identify  stakeholders  and  their  interests  and  perceptions,  include  them  in  the  study,  use  modelling 
primarily  as  a  means  to  clarify  issues  and  enhance  communication  but  also  to  identify  possible  options 
for  a  most  preferred  way  ahead  in  resolving  the  problematic  situation. 

>■  Messy  problems  cannot  be  fully  resolved  but  should  rather  be  managed. 

>  ‘Soft’  and  ‘hard’  OA  are  in  many  respects  complementary  to  one  another;  they  are  not  competing  OA 
approaches. 

A.2  CHAPTER  3  -  ACHIEVING  VALIDITY,  CREDIBILITY  AND 
ACCEPTANCE 

>  The  CoBP  identifies  how  validity,  credibility  and  acceptance  can  be  achieved  in  a  ‘soft’  OA  study. 

>■  The  primary  dimensions  of  validity  are  objectivity  and  rigour;  credibility  and  acceptance  are  not  distinct 
qualities  but  are  rather  derived  from  validity. 

>  The  analyst’s  aim  should  be  to  make  a  clear  distinction  between  the  reality  which  is  shared  amongst 
the  stakeholders,  and  the  sectional  perspectives  which  each  may  propagate  for  his  own  reasons. 

>  Gathering  subjective  judgements  from  experts  exposes  the  analyst  and  his  study  to  bias  which  can  be 
motivational  or  cognitive  in  nature. 

>  The  biggest  threat  to  validity  is  ignorance  at  the  design  stage  of  what  is  likely  to  be  important. 
This  threat  can  itself  be  mitigated  by  creating  an  iterative  design  similar  to  that  used  in  experimentation. 

>  An  analyst  needs  to  conduct  activities  in  an  ethical  manner  that  deserves  the  confidence  of  all  parties 
involved. 
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>  Credibility  and  acceptance  will  be  reinforced  by  independent  scrutiny. 

A.3  CHAPTER  4  -  ROLES  AND  RESPONSIBILITIES 

>  Many  roles  and  responsibilities  are  undertaken  during  the  conduct  of  a  ‘soft’  OA  study.  Moreover, 
one  individual  may  adopt  different  roles  involving  different  responsibilities.  Each  individual  involved 
in  the  study  contributes  to  its  quality  and  success. 

>  The  analyst,  through  his  design  and  conduct  of  the  study  and  through  his  reporting  and  interpretation 
of  the  study’s  outcomes,  plays  a  key  role  in  contributing  to  the  study’s  validity,  credibility  and  acceptance. 

>  The  facilitator  too,  through  his  management  of  study  process  (the  way  participants  interact),  plays  a 
key  role  in  contributing  to  the  study’s  validity,  credibility  and  acceptance. 

>  The  (primary)  client  has  a  responsibility  towards  the  analyst  in  supporting  the  execution  of  the  study 
and  should  act  as  a  partner  to  the  analyst. 

>  Good  stakeholder  management  is  the  responsibility  of  the  analyst  and  is  critical  to  study  success. 

A.4  CHAPTER  5  -  THE  ‘SOFT’  OA  STUDY  PROCESS 

□  Consider  designing  the  study  as  a  process  with  the  following  phases: 

•  Appreciation  -  ‘What  is  happening’? 

•  Analysis  -  ‘Why  and  how  is  it  happening’? 

•  Assessment  -  ‘How  can  it  happen  in  a  better  way’? 

•  Action  -  ‘What  needs  to  be  done’? 

□  Ensure  that  the  study  team  and  the  (group  of)  client(s)  reach  an  agreement  about  the  following  concepts 
at  the  end  of  each  of  the  4A-phases: 

•  Study  plan  (fr  Appreciation  phase) 

•  Problem  understanding  (  <r  Analysis  phase) 

•  Set  of  options  (4-  Assessment  phase) 

•  Action  plan  (4-  Action  phase) 

□  Be  aware  that  the  people  involved  in  the  process  belong  to  one  of  the  following  groups,  though  they 
should  cooperate  with  each  other: 

•  Study  Team:  group  of  people  who  ‘do  the  work’  (e.g.  the  facilitator,  the  analysts,  the  SMEs,  and  sub¬ 
sets  of  other  stakeholder  types  including  clients);  and 

•  Client(s):  group  of  decision  makers  (possibly  including  sponsors  and  customers). 

□  Be  aware  of  the  iterative  (cyclic)  and  dynamic  nature  of  the  process  and  its  phases. 

□  Recognise  the  divergent  or  convergent  nature  of  subsequent  phases  and  accommodate  for  it. 

□  Ensure  that  the  plan  for  application  of  ‘soft’  methods  is  understood  by  participants  and  clients  as  part 
of  their  programme  of  progressive  development.  The  same  goes  for  the  outputs  and  their  utility. 
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□  Provide  a  framework  which  allows  decision  makers  to  carry  through  their  decision-making  processes 
in  a  structured,  auditable  way. 

□  Create  a  roadmap  depicting  achievements  and  interrelations,  perhaps  in  a  facilitated  workshop. 

A.5  CHAPTER  6  -  DESIGNING  THE  ‘SOFT’  OA  STUDY  METHODOLOGY 

□  Communicate  with  client(s)  including  key  stakeholders  and  seek  agreement  on  any  assumption,  action 
and  delivery. 

□  Determine  the  most  likely  nature  of  the  problematic  situation  by  trying  to  shed  light  on  key  aspects. 

□  Decide  on  the  appropriateness  of  a  trajectory  from  perceived  mess  to  either  a  problem  proper  or  a 
manageable  mess. 

□  Identify  which  type(s)  of  uncertainty  exist(s)  and  contemplate  on  both  the  adequate  attitude  to  uncertainty 
and  ways  of  coping  with  it. 

□  Identify  stakeholders  and  their  interests  and  attitudes  towards  the  study  by  conducting  a  stakeholder 
analysis. 

□  Decide  on  the  appropriateness  of  a  single  or  a  multi-methodology  study  approach. 

□  Examine  all  methods  on  their  merits  to  the  problematic  situation  at  hand  and  decide  on  the 
appropriateness  of  specific  candidate  methods.  Recognise  own  limitations  in  knowledge/expertise  and 
seek  assistance. 

□  Decide  on  data  and  other  resources  needed  including  the  use  of  software. 

□  Be  aware  of  the  danger  of  dissipation  of  results  from  ‘hard’  OA  methods  when  interleaving  ‘hard’  and 
‘soft’  OA  methods  in  a  multi-methodology  approach. 

□  Maximise  objectivity  and  rigour,  gauging  the  validity  of  the  overall  approach  in  terms  appropriate  to 
the  problem  domain  being  addressed. 

□  Document  the  argumentation  and  rationale  for  modelling  decisions  and  keep  a  record. 

□  Document  the  argumentation  and  rationale  for  changes  in  the  study’s  assumptions  and  aim  (including 
aims  of  specific  study  phases),  the  definition  (or,  alternatively,  the  common  understanding)  of  the 
problematic  situation  in  order  to  ensure  end-to-end  integrity. 


A.6  CHAPTER  7  -  DATA  COLLECTION,  EXPERTS  AND  FACILITATED 
WORKSHOPS 

□  Understand  what  the  three  viewpoints  for  using  humans  as  data  sources  mean  for  the  study: 
the  cognitive,  critical  and  constructive  viewpoints.  Understand  types  of  bias  that  are  recognised  within 
each  perspective’s  context. 

□  Consider  carefully  what  stakeholders  (including  subject-matter  experts)  to  include  in  the  study,  and  how 
they  will  be  invited  to  participate. 

□  Design  facilitated  workshops  in  order  to  engage  with  stakeholders. 
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□  Select  the  key  conditions  in  the  Schuman  model  that  are  considered  to  be  critical  to  conducting 
workshops  in  the  current  study. 

□  Ensure  that  the  ‘theory  of  action’  underpinning  any  considered  course  of  action  is  understood  and 
documented. 

□  Design  (additional)  data  gathering  to  be  conducted  outside  a  workshop  setting,  including  what  sampling 
strategy  is  used. 

□  Consider  using  the  Human  Environment  Analysis  Reasoning  Tool. 

□  Document  all  data  collection  measures  taken  in  a  study,  including  the  methods,  the  sources  and  the 
collectors. 

A.7  CHAPTER  8  -  INTERPRETING,  USING  AND  COMMUNICATING 
RESULTS  AND  OUTCOMES 

□  Document  planned  method(s)  to  provide  a  reference  for  later  scrutiny  of  results. 

□  Conduct  triangulation  of  judgement,  using  documentation  and  empirical  data  where  possible  to  increase 
confidence  in  and  evidence  for  judgement. 

□  Determine,  in  co-operation  with  the  client,  where  bias  should  be  controlled  and  where  it  should  be 
measured  to  improve  understanding. 

□  Use  feedback  as  a  tool  to  understand  sensitivity  and  vested  interests. 

□  Give  careful  consideration  to  the  perceived  independence  of  the  analyst. 

□  Develop  a  clear  lexicon  by  which  outputs  of  the  study  can  be  described  to  the  client  community. 

□  Where  possible  include  the  client  or  his  stake  in  the  study  process  in  order  to  generate  ownership  of 
the  outcomes. 

□  Consider  how  learning  from  the  study  will  take  place  iteratively,  and  provide  outputs  that  support  this. 

□  Provide  capture  and  reporting  as  necessary  to  provide  traceability  and  transparency  in  the  methods  used. 
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This  annex  is  taken  from  a  UK  Dstl  Study  [1]  (with  slight  adaptation),  which  is  UK  ©  Crown  copyright 
2010.  It  is  re-published  here  with  the  permission  of  the  UK  Defence  Science  and  Technology’  Laboratory 
on  behalf  of  the  Controller  of  Her  Majesty’  ’s  Stationery  Office. 

As  an  example  of  ‘soft’  methods  in  use  in  a  defence  environment,  the  table  below  lists  the  ‘soft’  methods 
most  commonly  used  within  Dstl,  the  UK  Defence  Science  and  Technology  Laboratory  [1],  The  table  also 
indicates  the  types  of  analytic  activities  to  which  they  are  most  suited.  More  details  are  given  in  the  following 
sections,  taken  from  [1]  (but  textually  slightly  adapted).  Another  useful  source  of  recommendations  and 
techniques  for  defence  analysis,  including  ‘soft’  analysis,  is  [2], 


Table  B-1:  Some  ‘Soft’  OA  Methods  and  Their  Uses. 


No. 

Method 

Analytic  Activities 

1 

Benefits  modelling  (matrix  method) 

Option  selection 

2 

Benefits  modelling  (map-based) 

Problem  structuring  /  Option  generation  /  Option  selection 

3 

Cognitive  mapping 

Problem  structuring 

4 

Causal  mapping 

Problem  structuring 

5 

Influence  diagrams 

Problem  structuring 

6 

Decision  Trees 

Problem  structuring  /  Option  generation  /  Option  selection 

7 

SODA  (Strategic  Options  Development 
and  Analysis) 

Problem  structuring  /  Option  generation 

8 

Strategic  Choice  Analysis 

Problem  structuring  /  Option  generation 

9 

SSM  (Soft  Systems  Methodology) 

Problem  structuring 

10 

SAST  (Strategic  Assumptions  Surfacing 
and  Testing) 

Problem  structuring 

11 

SWOT  (Strengths,  Weaknesses, 
Opportunities,  Threats)  analysis 

Problem  structuring 

12 

PESTLEM  analysis  (Political,  Economic, 
Social,  Technological,  Legal, 
Environmental,  Military) 

Problem  structuring 

13 

Scenario  building/planning 

Problem  structuring  /  Option  generation 

14 

Interactive  Planning 

Problem  structuring  /  Gap  analysis  /  Concept  formulation 

15 

Robustness  analysis 

Option  generation  and  selection 

16 

Hypergames 

Problem  definition  /  Scenario  generation 

17 

Delphi 

Prioritisation  of  issues  /  Option  selection 

18 

AHP  (Analytic  Hierarchy  Process) 

Prioritisation  of  issues  /  Option  selection 

19 

Drama  theory  /  Confrontation  analysis 

Scenario  generation 

20 

Viable  systems  model 

Problem  structuring  /  Option  generation 

21 

Total  systems  intervention 

Problem  structuring  /  Planning  the  analysis 
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There  are  several  other  ‘soft’  methods  that  have  not  been  mentioned  in  [1]  and  Table  B-l  but  may  be 
useful  as  well,  including:  Morphological  analysis  (for  problem  structuring,  option  generation,  scenario 
building),  (qualitative)  System  Dynamics,  (for  problem  structuring,  time-dependent  multiple  perspective 
analysis,  option  generation  and  selection),  brainstorming  using  group  support  systems  (for  problem 
structuring,  option  generation),  several  other  (qualitative)  multiple  criteria  methods  (for  option  selection). 


B.l  BENEFITS  MODELLING  (MATRIX  METHOD) 

Benefits  modelling  is  a  flexible  technique  by  which  the  improvements  in  an  enabling  technology  (such  as 
CIS)  can  be  systematically  and  quantifiably  related  to  improvements  in  the  overall  effectiveness  of  the 
organisation  which  the  technology  supports.  The  key  philosophy  behind  the  approach  is  to  decompose  the 
problem  into  a  number  of  distinct  steps,  each  of  which  represents  a  concept  that  can  be  directly  related  to 
the  real  world.  Decomposition  of  the  problem  in  this  way  to  form  a  ‘benefits  chain’  has  many  advantages 
as  it  allows  individual  aspects  of  the  problem  to  be  considered  separately.  However,  the  benefits  chain 
forms  a  static  model  which  does  not  allow  any  form  of  feedback  to  earlier  stages. 

Benefits  modelling  is  a  multi-criteria  analysis  technique,  in  that  multiple  measures  can  be  used  at  each 
stage  of  the  benefits  chain,  including  the  final  measures  of  effectiveness.  Traditionally  the  data  needed  to 
populate  a  benefits  model  have  been  obtained  from  technical  and  military  judgement.  The  use  of  a  benefits 
chain  facilitates  the  use  of  judgmental  techniques  in  situations  where  a  wide  range  of  types  of  experience 
is  required.  However,  the  benefits  model  technique  is  not  restricted  to  using  judgmental  data.  Data  could 
be  derived  from  the  results  of  lower-level  objective  models.  Note  that  judgements  are  used  directly  in  the 
model  to  quantify  improvements;  this  is  in  contrast  to  techniques  such  as  the  Analytical  Hierarchy  Process 
(AHP)  where  judgements  are  used  to  derive  weights  through  some  form  of  pair-wise  comparison. 

The  benefits  chain  is  constructed  from  a  series  of  dependency  matrices  each  of  which  represents  the 
relationship  between  one  set  of  concepts  (often  referred  to  as  an  axis)  and  those  relevant  to  the  next  step  in 
the  chain.  The  matrix  transitions  typically  relate  CIS  system  performance,  through  intermediate  measures 
(such  as  measures  of  C2I  function  performance),  to  high-level  measures  of  effectiveness  which  quantify 
how  well  warfare  (and  OOTW)  functions  can  be  performed.  A  generic  structure  might  have  five  axes  as 
follows: 


•  Investments  -  Meaningful  units  of  investment  in  some  aspect  of  the  system-of-systems,  including 
equipment  and  training. 

•  System  Functions  and  Attributes  -  Primarily  related  to  the  performance  of  technical  elements  of 
the  system,  but  could  include  human  skills  if  the  investment  is  in  training. 

•  Organisation  Activities  and  Attributes  -  Related  to  the  combined  socio-technical  system, 
and  covering  activities  such  as  information  processing  and  attributes  such  as  flexibility. 

•  Military  Capabilities  and  Attributes  -  The  outputs  of  the  organisation;  the  activities  or  attributes 
that  are  necessary  for  achieving  a  valuable  outcome. 

•  Value  System  -  Those  concepts  that  define  a  successful  campaign. 


B.2  BENEFITS  MODELLING  (MAP-BASED) 

Benefits  analysis  is  a  technique  which  arose  in  the  1990s,  combining  elements  of  Multi-Criterion  Decision 
Analysis  (MCDA)  and  causal  mapping.  It  aims  to  model  the  consequences  of  investment  decisions  in 
terms  of  their  impact  on  operational  effectiveness. 

The  construction  of  the  causal  map  often  (but  not  invariably)  starts  with  the  identification  of  the  desired 
end  benefit  of  the  programme.  The  map  is  laid  out  so  that  the  means  precede  the  end  -  so,  for  each  effect, 


B  -2 


RTO-TR-SAS-087 


ANNEX  B  -  SELECTIVE  LIST  OF  ‘SOFT’  OA  METHODS 


all  possible  causes  of  that  effect  must  be  identified.  This  process  is  iterated  until  no  further  decomposition 
of  causality  can  be  achieved;  the  final  causes  are  the  inputs  to  the  model,  against  which  (in  this  case) 
investment  options  are  to  be  scored.  This  mapping  process  can  be  very  helpful  in  establishing  in  broad 
terms  whether  all  aspects  impacting  on  capability  have  been  covered  by  a  requirements  document. 

Once  the  causal  map  has  been  developed,  and  a  stable  state  reached,  it  can  be  used  as  the  basis  for  MCDA 
analysis.  In  generic  terms,  the  most  fundamental  level  in  the  causal  map  will  need  to  be  measured 
numerically  and  scored,  and  then  the  mathematical  relationships  between  each  layer  of  the  map  formalised 
and  weighted.  This  will  permit  the  maps  to  be  used  in  quantitative  analysis. 

B.3  COGNITIVE  MAPPING 

This  technique  maps  the  vision  of  a  system,  issue  or  event  from  the  point  of  view  of  an  individual  (‘group 
map’  or  ‘concept  map’  is  the  term  usually  used  when  reflecting  the  views  of  a  group  of  people).  It  captures 
the  interpretation  of  how  concepts  relating  to  the  area  of  interest  interact.  The  cognitive  map  can  be  used 
to  analyse  the  knowledge  and  opinions  of  the  stakeholder(s). 

Generally,  cognitive  maps  used  in  ‘soft’  OA  are  constrained  to  be  causal  maps,  representing  cause  and 
effect  between  the  various  concepts  in  the  problem  space.  The  process  of  developing  the  map  will 
generally  start  with  a  less  formal  structure  and  the  map  will  be  refined  over  the  course  of  the  workshop. 

The  language  used  in  a  cognitive  map  is  chosen  carefully.  For  example,  in  the  SODA  approach  concepts  are 
phrased  as  imperatives  (commands  to  carry  out  an  action).  In  general,  the  manner  of  phrasing  a  concept 
should  be  consistent  throughout  the  finished  map.  However,  during  the  mapping  process,  facilitators  should 
be  observant  of  the  language  used  by  stakeholders  to  express  the  concepts,  which  can  be  significant. 

Cognitive  mapping  is  a  way  of  improving  understanding  of  a  problem  area,  and  can  be  particularly  useful 
in  teasing  out  the  different  viewpoints  of  various  stakeholders.  It  is  particularly  useful  at  the  scoping  phase 
of  a  study. 

B.4  CAUSAL  MAPPING 

This  is  a  variant  of  cognitive  mapping,  which  imposes  the  discipline  that  the  elements  on  the  map  must 
form  a  causal  sequence.  Causal  maps  can  be  used  as  the  basis  for  a  benefits  model  by  representing  the 
impacts  of  study  interventions  on  operational  effectiveness. 

Causal  maps  can  be  constructed  top-down  or  bottom-up.  There  are  advantages  to  each  method,  depending 
on  how  well  the  problem  is  understood.  The  top-down  approach  requires  the  goal  to  be  clearly  defined, 
and  imposes  a  strict  discipline;  at  each  stage  the  question  “HOW  does  this  concept  cause  the  next?”  must 
be  asked. 

Where  the  map  is  representing  a  well-understood  system,  the  inputs  (variables  or  interventions)  can  be 
used  as  the  starting  point  and  the  consequences  inferred.  Whichever  method  is  used  to  construct  the  map, 
it  is  important  to  cross-check  using  the  alternative  route  -  for  each  concept  asking  “HOW  might  this  be 
achieved?”  and  “WHAT  are  the  consequences  of  this?” 

B.5  INFLUENCE  DIAGRAMS 

Influence  diagrams  are  simple  representations  of  problems,  including  decisions,  uncertainties  and  aims 
and  how  they  influence  each  other.  They  are  particularly  useful  in  the  early  phases  of  structuring  big  and 
complex  problems,  as  they  enable  the  relationships  between  the  main  elements  of  the  problem  to  be  identified. 
They  are  often  used  together  with  decision  trees. 
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B.6  DECISION  TREES 

Decision  trees  are  used  to  provide  a  diagrammatic  representation  of  a  problem,  setting  out  possible  courses 
of  action  and  their  consequences.  They  show  the  details  of  a  problem  and  can  provide  a  means  of  quantifying 
and  evaluating  options. 


B.7  STRATEGIC  OPTIONS  DEVELOPMENT  AND  ANALYSIS  (SODA) 
(RENAMED  TO:  JOURNEY  MAKING) 

SODA  is  most  suited  to  complex,  ill-defined  problems  with  multiple  stakeholders  with  differing  views  of 
the  problem  space.  It  captures  the  various  views  and  provides  the  opportunity  for  the  stakeholders  to  reach 
a  shared  understanding  of  the  problem. 

SODA  uses  individual  and  group  Cognitive  Maps  (see  above)  to  explore  a  problem  space.  This  leads  to  the 
identification  and  negotiation  of  possible  solution  options  in  facilitated  workshops  using  the  Oval  Mapping 
technique.  The  technique  aims  to  create  a  causal  map  with  a  “teardrop”  shape,  with  one  or  a  small  number  of 
goals,  supported  by  a  greater  number  of  issues  or  strategies  which  are  in  turn  supported  by  more,  detailed 
options. 

A  successful  outcome  from  SODA  would  be  the  achievement  of  consensus  among  the  stakeholders  on  a 
set  of  possible  viable  courses  of  action.  It  would  not  seek  to  define  the  optimal  solution,  but  could  be  used 
as  a  precursor  for  MCDA. 


B.8  STRATEGIC  CHOICE  ANALYSIS  (SCA) 

This  method  is  related  to  interactive  planning.  It  sets  out  to  establish  and  manage  the  areas  of  uncertainty 
in  a  problem,  through  collaboration  between  stakeholders.  It  considers  all  the  decisions  involved  in  a 
particular  situation  in  parallel,  rather  than  taking  them  according  to  perceived  priority  or  timescale,  which 
permits  the  impacts  of  immediate  actions  of  future  decisions  to  be  explored. 

Three  key  areas  of  analysis  are  used  to  structure  the  problem: 

•  The  decision  area,  where  alternative  courses  of  action  are  being  considered. 

•  The  comparison  area,  where  the  alternatives  are  assessed  and  ranked. 

•  The  uncertainty  areas,  where  the  following  issues  are  investigated: 

•  Working  environment; 

•  Guiding  values  (policies);  and 

•  Related  agendas  (other  perspectives). 

Four  complementary  areas  of  decision  making  are  considered  in  SCA: 

•  Shaping,  where  the  problem  structure  and  linkages  are  considered; 

•  Designing,  where  potential  courses  of  action  are  explored; 

•  Comparing,  where  the  criteria  for  assessing  courses  of  action  are  decided;  and 

•  Choosing. 

The  key  to  this  approach  is  in  the  consideration  of  the  whole  problem  space,  which  ensures  that  options 
generated  for  different  parts  of  the  problem  will  be  compatible.  The  effectiveness  of  SCA  is  heavily 
dependent  on  the  skill  of  the  facilitator. 
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B.9  SOFT  SYSTEMS  METHODOLOGY  (SSM) 

Soft  Systems  Methodology  helps  to  structure  complex  problems  where  there  may  be  divergent  views 
about  the  nature  and  definition  of  the  problem.  It  is  particularly  useful  for  requirements  capture.  It  uses 
‘real-world’  descriptions  and  systems  engineering  terms  to  ensure  both  that  the  problem  is  fully  described 
and  that  it  can  be  rigorously  modelled. 

The  main  elements  of  SSM  are  as  follows: 

•  Through  facilitated  workshops  and  interviews  with  the  problem  stakeholders,  a  “rich  picture”  is 
created,  using  “real-world”  terms  which  encourages  the  issues  to  be  explored  creatively  and  ensures 
that  all  relevant  factors  are  considered. 

•  Using  the  rich  picture,  the  analyst  derives  the  root  definition  of  the  problem  using  the  mnemonic 
“CATWOE”  (see  below)  to  ensure  that  all  relevant  aspects  of  the  problem  are  included: 

C  -  Customer; 

A  -  Actor; 

T  -  Transformation; 

W  -  World  view  (Weltanschauung); 

O  -  Owner;  and 

E  -  Environmental  constraints. 

•  The  root  definitions  are  used  to  create  conceptual  models  of  the  activities  implied  by  the  root 
definition. 

•  These  models  are  then  compared  with  the  real  world. 

•  Potential  interventions  and  their  likely  impacts  can  then  be  defined. 

B.10  STRATEGIC  ASSUMPTIONS  SURFACING  AND  TESTING  (SAST) 

Strategic  Assumptions  Surfacing  and  Testing  is  a  method  used  to  bring  consensus  to  a  group  of 
stakeholders  where  differing  views  are  preventing  them  from  making  progress  in  dealing  with  a  problem. 
As  the  name  suggests,  it  draws  out  the  underlying  assumptions  held  by  the  stakeholders  and  clarifies  the 
common  and  opposing  issues. 

It  is  a  five-stage  process;  the  stages  are  described  below. 

•  Group  Formation  -  The  stakeholders  are  formed  into  mixed  groups  (chosen  to  provide  a  range 
of  roles  and  experiences  but  also  to  minimise  conflict)  and  each  group  is  given  a  “viewpoint”  from 
which  to  address  the  issue. 

•  Assumption  Surfacing  -  Each  group  brainstorms  the  issue  from  their  specific  viewpoint  and 
identifies  the  relevant  assumptions. 

•  Intra-group  Dialectic  Debate  -  At  this  stage  each  assumption  is  tested  for  importance  by  examining 
its  opposite,  and  for  certainty  according  to  the  group’s  best  estimate.  Assumptions  that  are  deemed 
to  be  neither  certain  nor  important  are  dropped  at  this  stage;  those  which  are  both  certain  and 
important  are  retained  and  the  others  ranked  within  the  group  (importance  is  generally  rated  more 
highly  than  certainty). 

•  Inter-group  Dialectic  Debate  -  Each  group  presents  its  findings  before  a  whole  group  discussion  is 
held.  The  output  of  this  stage  is  a  set  of  agreed  assumptions  (which  will  be  taken  as  underpinning 
principles  on  which  to  proceed)  and  a  list  of  those  requiring  further  debate. 
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•  Final  Synthesis  -  The  whole  group  works  on  the  outstanding  controversial  issues  to  achieve 
consensus.  The  final  output  is  a  prioritised  list  of  the  most  important  issues,  an  assessment  of  the 
status  of  potential  solutions  to  the  issues  and  a  proposal  for  the  next  steps. 

This  method  is  particularly  useful  in  developing  plans  from  a  high-level  strategy. 

B.ll  SWOT  (STRENGTHS,  WEAKNESSES,  OPPORTUNITIES,  THREATS) 
ANALYSIS 

This  method  provides  a  structure  for  brainstorming  a  problem  in  its  early  stages.  It  is  particularly  helpful 
in  generating  solution  options.  The  TOWS  technique  extends  this  by  pairing  the  individual  elements. 


B.12  PESTLEM  ANALYSIS  (POLITICAL,  ECONOMIC,  SOCIAL, 

TECHNOLOGICAL,  LEGAL,  ENVIRONMENTAL,  MILITARY) 

Again,  this  method  provides  a  structure  for  brainstorming  workshops,  in  this  case  examining  the  system 
constraints  and  operating  environment. 


B.13  SCENARIO  BUILDING/PLANNING 

This  method  is  intended  to  achieve  a  better  understanding  of  the  future  environment.  It  is  a  group  process 
which  draws  out  an  understanding  of  the  key  issues  affecting  the  future.  The  aim  is  to  produce  a  number 
of  diverging  stories  by  extrapolating  the  main  driving  forces.  This  should: 

•  Increase  the  knowledge  and  understanding  of  the  problem  environment;  and 

•  Widen  the  perception  of  possible  future  events. 

The  method  involves  a  5 -step  process: 

•  Identify  the  key  issue; 

•  Identify  the  primary  driving  forces  affecting  the  issue  at  present  (using  the  PEST  categories); 

•  Identify  the  predetermined,  fixed  elements  of  these  driving  forces; 

•  Identify  the  critical  uncertainties  and  cluster  them  so  that  two  orthogonal  axes  of  uncertainty  are 
defined,  and  create  a  matrix  defining  four  quadrants  of  uncertainty.  Each  of  these  quadrants  can 
then  define  a  possible  future  scenario;  and 

•  Use  the  main  drivers  to  flesh  out  the  scenarios  defined  by  the  four  quadrants. 

This  method  provides  a  set  of  four  contrasting  potential  future  scenarios  against  which  future  plans  can  be 
tested. 

B.14  INTERACTIVE  PLANNING 

This  technique  is  associated  with  process  change  programmes,  where  an  organisation  needs  to  evolve. 
It  is  a  three-step  method: 

•  Mess  Formulation  -  Where  multiple  interdependent  problems  exist,  the  first  stage  is  to  focus  on 
the  current  business  situation  and  analyse  the  function,  structure  and  critical  processes  of  the 
organisation.  This  analysis  will  help  to  identify  the  current  obstacles  to  meeting  the  organisation’s 
goals,  from  which  a  reference  scenario  can  be  generated  describing  the  worst-case  consequences 
of  retaining  the  status  quo. 
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•  Design  Process  -  This  step  requires  blue-skies  thinking  to  create  an  unconstrained  vision  of  the 
future  organisation,  which  can  then  be  iteratively  refined  so  that  the  vision,  mission,  structure  and 
process  can  be  defined.  The  reference  scenario  is  used  to  check  the  design  to  ensure  that  the 
obstacles  and  problems  identified  in  the  first  stage  can  be  handled. 

•  Means  Planning  -  Where  the  strategy  and  tactics  to  achieve  the  new  organisation  are  developed. 

The  strength  of  Interactive  Planning  is  the  clearly  structured  process  which  starts  with  the  identification  of 
the  current  barriers  to  success  before  attempting  to  design  the  new  organisation. 


B.15  ROBUSTNESS  ANALYSIS 

Robustness  analysis  looks  at  decision-making  in  conditions  of  considerable  uncertainty.  It  requires  the 
analyst  to  acknowledge  that  it  is  not  possible  to  predict  the  future,  or  event  to  estimate  the  possibilities  of 
future  events,  but  instead  one  should  use  decision  trees  to  look  at  all  possible  futures  and  consider  their 
desirability. 

This  technique  uses  only  the  most  reliable  information  about  the  decision  tree  -  the  number  of  branches  at 
any  point  -  and  does  not  look  at  the  probabilities  of  any  of  the  events;  effectively,  it  considers  all  futures 
as  equally  likely.  The  robustness  of  a  decision  option  is  defined  as  the  number  of  acceptable  futures  at  the 
planning  horizon  with  which  the  option  is  compatible  (as  a  percentage  of  the  total  number  of  acceptable 
futures  at  the  planning  horizon). 


B.16  HYPERGAMES 

Hypergames  build  on  classical  game  theory  to  give  a  method  of  more  practical  use  by  recognising  that  the 
players  (two  or  more)  have  different  perceptions  of  the  game  being  played.  This  representation  of  different 
perceptions  is  important  in  judging  the  value  of  different  strategies  to  different  players,  and  hence  this 
approach  can  be  used  to  define  (and  subsequently  refine)  a  solution  space  for  a  complex  problem.  Because 
of  the  increased  complexity  of  hypergames  compared  with  classical  gaming  methods,  families  of  decision 
trees  rather  than  matrices  are  used. 


B.17  DELPHI 

Delphi  is  a  term  applied  to  the  collection  of  expert  judgements  from  a  group  of  people.  In  particular, 
the  information  is  not  attributable  to  any  individual;  it  is  collected  anonymously  and  discussed  without 
reference  to  individual  opinions.  Usually  the  data  gathering  will  involve  the  use  of  questionnaires  and  secret 
voting,  followed  by  generalised  feedback  and  discussion  sessions  before  iterating  the  process.  The  aim  is 
to  generate  consensus  without  direct  influence  or  coercion  by  any  individual. 

Numerous  methods  of  implementing  this  method  have  been  developed,  and  in  particular,  where  the  aim  is 
to  rank  or  score  decision  options  a  number  of  different  scoring  systems  exist. 


B.18  ANALYTICAL  HIERARCHY  PROCESS  (AHP  OR  SAATY  METHOD) 

AHP  is  a  multi-criteria  decision  making  method  widely  used  in  support  to  defence  acquisition.  It  is 
designed  to  assist  in  ranking  solutions  to  complex  problems.  The  process  can  be  broken  into  four  stages: 

•  Problem  structuring  as  a  hierarchy  of  criteria  and  options; 

•  Criteria  weighting; 
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*  Option  scoring;  and 

•  Sensitivity  analysis. 

The  first  stage  identifies  the  evaluation  criteria;  the  attributes  by  which  the  solution  options  will  be  judged. 
In  more  complex  problems,  these  criteria  can  be  subdivided  and  a  hierarchical  structure  of  criteria  will 
ensue. 

The  next  stage  is  to  weight  these  criteria  in  importance  and  thus  prioritise  issues.  These  weightings  are 
subjective;  a  number  of  techniques  exist  for  deriving  these  weights.  One  of  the  more  commonly  used 
methods  is  pair-wise  comparison,  carried  out  at  each  node  of  the  hierarchy.  More  detail  on  this  can  be 
found  in  the  references  of  [1]. 

The  third  stage  is  to  score  each  solution  option  against  each  criterion.  Some  criteria  can  be  scored  objectively, 
with  measurable  data;  some  criteria  are  subjective.  It  is  possible  to  quantify  subjective  judgements  using  the 
same  numerical  scale  as  for  objectively  derived  scores.  Again,  this  is  explained  in  detail  in  the  references  of 
[!]• 

Finally,  sensitivity  analysis  should  be  carried  out.  This  is  important  in  hierarchical  models.  The  impact  on 
the  results  of  small  variations  in  the  input  data  (scores  and  weights)  must  be  investigated  to  assess  the 
robustness  of  the  model. 

B.19  DRAMA  THEORY  (ALSO  KNOWN  AS  CONFRONTATION  ANALYSIS) 

Confrontation  analysis  also  has  its  basis  in  game  theory.  This  technique  is  less  formal  than  hypergaming; 
it  focuses  on  the  tactical,  political  and  psychological  relationships  between  the  players,  and  looks  at  the 
players’  beliefs  about  the  other  players’  intentions. 

The  first  stage  in  the  process  is  to  define  the  present  situation,  the  predicted  future  situation  that  would 
arise  if  current  plans  were  implemented,  each  player’s  preferred  outcome  and  potential  areas  for  conflict 
and  compromise.  The  next  stage  involves  the  analysis  of  threats  and  promises;  how  one  player’s 
impression  of  another  player’s  intentions  affect  his  plans.  The  culmination  of  this  process  is  a  strategic 
map  which  sets  out  all  players’  possible  courses  of  action  to  reach  their  preferred  endpoint.  This  can  then 
be  used  to  identify  courses  which  lead  to  a  desired  outcome. 


B.20  VIABLE  SYSTEMS  MODEL 

This  model  is  applicable  to  organisational  change  programmes.  It  proposes  five  necessary  functions  which 
must  exist  for  an  organisation  to  be  viable: 

•  Primary  activities; 

•  Coordination  between  the  primary  activities; 

•  Control; 

•  Intelligence;  and 

•  Policy. 

The  application  of  this  model  involves  the  critical  examination  of  each  of  these  functions  by  the  key 
stakeholders.  Workshops  are  held  and  the  current  status  and  future  goals  are  established.  When  the  gaps 
have  been  identified,  plans  to  realise  the  goals  can  be  generated. 
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B.21  TOTAL  SYSTEMS  INTERVENTION 

This  is  a  method  for  characterising  problems  and  mapping  them  to  appropriate  analysis  methodologies. 
It  is  a  three-phased  approach,  starting  with  data  gathering  from  a  wide  range  of  stakeholders,  then 
classifying  the  problem  type  and  finally  implementing  the  appropriate  methodology  for  its  solution. 


B.22  REFERENCES 

[1]  Handley,  A.,  “Guidance  on  the  use  of  subjective  Operational  Analysis  methods  in  support  of  acquisition 
decisions”,  DSTL/CR43706,  March  2010.  ©  Crown  copyright  2010. 

[2]  NATO  Joint  Analysis  and  Lessons  Learned  Centre  (Lisbon,  Portugal;  www.jalcc.nato.int),  “Joint 
Analysis  Handbook”,  3rd  Ed.,  2007. 
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